JP6337911B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, swinging in tennis, golf, baseball, or the like is photographed with a camera, and the form is checked while viewing the photographed swing image (for example, Patent Document 1).

  In addition, when checking such a form, there is also one that facilitates the check by displaying the locus of a moving body such as a golf club (for example, Patent Document 2).

JP 2003-1117045 A JP-A-11-98450

  By the way, you can check the form not only by taking a picture of your swing with a camera, but also by taking a picture of a swing with an ideal form such as a professional player or instructor, and comparing it with this to check it. May do.

  However, in such a case, the shooting conditions such as the angle of the camera that shoots the swing with the ideal form do not always coincide with those when the swing is shot. Even if the described technique is used, there is a problem that an ideal form cannot be accurately grasped and it is difficult to check the form.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of accurately grasping an ideal form even if photographing conditions are different.

In order to solve the above problems, an image processing apparatus according to the present invention provides:
Second moving image object that can be moved, which is taken under a different second imaging environment from the at least one first moving image and the first photographing environment of the shooting under the first imaging environment Image acquisition means for acquiring
A trajectory generating means for generating a first trajectory based on the motion of the subject included in the first moving image acquired by the image acquiring means;
Position information acquisition means for acquiring position information in the image based on feature points included in the movable subject of the first moving image and the second moving image;
A trajectory correcting unit that corrects the first trajectory generated by the trajectory generating unit based on the positional information acquired by the positional information acquiring unit so as to fit in the second imaging environment;
And synthesizing for synthesizing means such that the first trajectory is superimposed corrected by the trajectory compensation means to the second moving image,
It is provided with.

  According to the present invention, an ideal form can be accurately grasped even if photographing conditions are different.

1 is a diagram illustrating an overall configuration of an image processing system. It is a figure which shows the internal structure of an image processing apparatus. It is a figure which shows the functional structure of a control part. It is a flowchart explaining the procedure of displaying or analyzing the locus | trajectory of a moving body. It is a flowchart explaining a data registration process. It is a figure explaining the designation | designated screen of a form check start frame. It is a figure explaining the screen which designates a line. It is a figure explaining the screen which designates a swing operator's position. It is a figure explaining the screen which designates the position of a tennis racket. It is a figure explaining the screen which designates the position of a tennis racket. It is a figure explaining the designation | designated screen of a backswing start frame. It is a figure explaining the designation | designated screen of a follow swing start frame. It is a figure explaining the designation | designated screen of a follow swing end frame. It is a figure explaining the inclination of the defined line. It is a figure explaining calculation of a photography angle. It is a figure explaining calculation of a photography angle. It is a figure explaining the locus | trajectory of a tennis racket. It is a figure explaining the moving image of a check object. It is a figure explaining a file selection screen. It is a figure explaining correction | amendment of a locus | trajectory. It is a figure explaining correction | amendment of a locus | trajectory. It is a figure explaining the reproduction | regeneration timing of a moving image. It is a figure explaining the diagnostic screen of a check subject's form. It is a figure explaining the designation | designated screen of a form check start frame. It is a figure explaining the screen which designates a line. It is a figure explaining the screen which designates a swing operator's position. It is a figure explaining the screen which designates the position of a golf club. It is a figure explaining the screen which designates the position of a golf club. It is a figure explaining the designation | designated screen of a backswing start frame. It is a figure explaining the designation | designated screen of a follow swing start frame. It is a figure explaining the designation | designated screen of a follow swing end frame. It is a figure explaining the inclination of the defined line. It is a figure explaining calculation of a photography angle. It is a figure explaining calculation of a photography angle. It is a figure explaining calculation of a photography angle. It is a figure explaining calculation of a photography angle. It is a figure explaining correction | amendment of a locus | trajectory. It is a figure explaining correction | amendment of a locus | trajectory. It is a figure explaining the diagnostic screen of a check subject's form. It is a figure explaining the screen which designates a golf club and the position of a shoulder of a swing operator. It is a figure explaining the screen which designates a golf club and the position of a shoulder of a swing operator. It is a figure explaining the setting of a locus. It is a figure explaining the locus | trajectory of a golf club.

FIG. 1 shows an overall configuration of an image processing system 100 according to an embodiment of the present invention.
The image processing system 100 is a system for displaying or analyzing a trajectory of a moving object. In the present embodiment, as an example, a case where a trajectory of a tennis swing on a tennis court is displayed or analyzed will be described. According to the present embodiment, it is possible to display or analyze a golf swing, a bat swing in baseball, and other moving object trajectories.

  The image processing system 100 includes cameras 1 to 3 as imaging means, a HUB 4, an image processing apparatus 10, a printer 5, a media writing unit 6, and the like. These are connected by a LAN (Local Area Network) cable 7.

  Cameras 1 to 3 are high-function digital cameras that capture about 30 still images per second and output moving images. The number of frames captured per second (frame rate) is represented by fps (frame per second). That is, the frame rate of the cameras 1 to 3 is 30 [fps]. In the present embodiment, as described above, a camera with a frame rate of 30 [fps] is applied. However, the frame rate is not limited to this, for example, a high-speed camera with a frame rate of 300 to 1200 [fps]. May be applied.

  The camera 1 is a camera that shoots the swing operator at a first shooting angle with respect to the back line of the tennis court, and outputs image data of a moving image shot at the position to the image processing apparatus 10. The camera 2 is a camera that shoots the swing operator at a second shooting angle with respect to the back line of the tennis court, and outputs image data of a moving image shot at the position to the image processing apparatus 10. The camera 3 is a camera that shoots the swing operator at a third shooting angle with respect to the back line of the tennis court, and outputs image data of a moving image shot at the position to the image processing apparatus 10.

  The image processing apparatus 10 inputs image data of moving images taken by the cameras 1 to 3 and performs display or analysis of moving images based on the input image data.

FIG. 2 shows an internal configuration of the image processing apparatus 10.
The image processing apparatus 10 includes a control unit 11, an input unit 12, a display unit 13, a storage unit 14, a connection unit 15, and the like. Each unit is connected by a bus 16. In the present embodiment, the image processing apparatus 10 is configured by an information processing apparatus such as a personal computer, for example. Note that the camera 1 to 3 may have the functions of the image processing apparatus.

  The control unit 11 that functions as a trajectory correction unit includes a CPU, a RAM, and the like (not shown), and centrally controls each unit of the image processing apparatus 10 in cooperation with the storage unit 14. That is, the control unit 11 executes the various programs stored in the storage unit 14 to generate a first setting unit 11a, a second setting unit 11b, a first trajectory acquisition unit 11c, and a second track, as illustrated in FIG. Trajectory acquisition unit 11d, first shooting angle calculation unit 11e, second shooting angle calculation unit 11f, shooting angle difference calculation unit 11g, correction unit 11h, third setting unit 11i, fourth setting unit 11j, ratio calculation unit 11k, position It functions as a difference calculation unit 11m, a trajectory difference determination unit 11n, an image designation unit 11p, a movement target image extraction unit 11r, and a display control unit 11s.

  The first setting unit 11a sets two or more straight lines that intersect as first feature points in a moving image obtained by photographing with the cameras 1 to 3 under the first photographing environment. Although specifically described later, the first setting unit 11a sets two or more straight lines by the user based on a moving image obtained by photographing a swing with an ideal form under a certain photographing environment. For example, in a tennis court, these two or more straight lines are straight lines set respectively along the back line and the side line.

  The second setting unit 11b sets two or more straight lines that intersect as the second feature points in the moving image obtained by shooting with the cameras 1 to 3 in a second shooting environment different from the first shooting environment. To do. Specifically, as described later, the second setting unit 11b allows the user to set two or more by the user based on the moving image obtained by photographing the check target swing under a photographing environment different from that when photographing the ideal form. Set a straight line. For example, in a tennis court, these two or more straight lines are straight lines set respectively along the back line and the side line.

  The 1st locus | trajectory acquisition part 11c acquires the locus | trajectory of a 1st movement object as a 1st locus | trajectory from the moving image obtained by image | photographing on the 1st imaging environment with the cameras 1-3. Although specifically described later, the first trajectory acquisition unit 11c acquires a trajectory of a movement target such as a tennis racket or a golf club head from a captured moving image of an ideal form.

  The second trajectory acquisition unit 11d acquires the trajectory of the second movement target as the second trajectory from the moving image obtained by photographing with the cameras 1 to 3 in the second photographing environment. Although specifically described later, the second trajectory acquisition unit 11d acquires a trajectory of a movement target such as a tennis racket or a golf club head from the captured moving image of the swing to be checked.

  The first shooting angle calculation unit 11e calculates the shooting angles of the cameras 1 to 3 when shooting is performed in the first shooting environment from angles formed by two or more straight lines set by the first setting unit 11a. . Specifically, as will be described later, the first shooting angle calculation unit 11e is a camera for a certain position based on an angle formed by each of two or more straight lines set from a moving image obtained by shooting a swing with an ideal form. The shooting angles of 1 to 3 are calculated.

  The second shooting angle calculation unit 11f calculates the shooting angles of the cameras 1 to 3 when shooting is performed in the second shooting environment from the angles formed by the two or more straight lines set by the second setting unit 11b. . As will be described in detail later, the second shooting angle calculation unit 11f is configured to detect the camera 1 with respect to a certain position from angles formed by two or more straight lines set from a moving image obtained by shooting the swing to be checked. The imaging angle of ~ 3 is calculated.

  The shooting angle difference calculation unit 11g calculates a difference between the shooting angle calculated by the first shooting angle calculation unit 11e and the shooting angle calculated by the second shooting angle calculation unit 11f.

  The correction unit 11h corrects the position of the first trajectory acquired by the first trajectory acquisition unit 11c based on the difference in the shooting angle calculated by the shooting angle difference calculation unit 11g. As will be described in detail later, the correction unit 11h is configured so that the shooting angle of the cameras 1 to 3 when shooting is performed in the first shooting environment and that of the cameras 1 to 3 when shooting is performed under the second shooting environment. Based on the difference from the shooting angle, the position of the trajectory of the movement target such as a tennis racket or a golf club head acquired from the captured moving image of the ideal form is corrected.

  The 3rd setting part 11i sets the straight line as a 3rd feature point in the moving image obtained by image | photographing on the 1st imaging environment with the cameras 1-3. Although specifically described later, the third setting unit 11i sets a straight line different from the straight line as the first feature point by the user based on a moving image obtained by photographing a swing with an ideal form. . This straight line is, for example, a straight line set along the center line of the body of the swing operator performing a swing with an ideal form.

  The fourth setting unit 11j sets a straight line as the fourth feature point in the moving image obtained by shooting with the cameras 1 to 3 in the second shooting environment. Although specifically described later, the fourth setting unit 11j sets a straight line different from the straight line as the second feature point by the user based on the moving image obtained by photographing the swing to be checked. This straight line is, for example, a straight line set along the center line of the body of the swing operator who performs the check target swing.

  The ratio calculation unit 11k calculates the ratio of the length of the straight line set by the third setting unit 11i and the straight line set by the fourth setting unit 11j.

  The position difference calculation unit 11m calculates a position difference between the straight line set by the third setting unit 11i and the straight line set by the fourth setting unit 11j. That is, the position difference calculation unit 11m obtains a deviation amount between the position of the straight line as the third feature point and the position of the straight line as the fourth feature point.

  The trajectory difference determination unit 11n determines a difference between the first trajectory corrected by the correction unit 11h and the second trajectory acquired by the second trajectory acquisition unit 11d. Although specifically described later, the trajectory difference determination unit 11n captures the ideal form in which the position of the trajectory of the movement target such as the tennis racket or the golf club head is corrected from the captured moving image of the check target swing. It is determined how much from the trajectory of the movement target such as a tennis racket or a golf club head acquired from the swing moving image by the above.

  The image designation unit 11p designates the first movement target image. Although specifically described later, the image designation unit 11p designates an image of the movement target portion by the user in order to specify the movement target.

  The movement target image extraction unit 11r extracts an image corresponding to the image designated by the image designation unit 11p from the moving images obtained by photographing in the first photographing environment with the cameras 1 to 3. Although specifically described later, the movement target image extraction unit 11r performs template matching using an image designated by the user as a template. Thereby, an image corresponding to the template is extracted from each still image constituting the moving image obtained by photographing the swing with the ideal form.

  The display control unit 11s performs display control of the display unit 13.

  The input unit 12 includes a keyboard having a cursor key and various function keys, a mouse, and the like, and outputs an operation signal to the control unit 11.

  The display unit 13 is configured by an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or the like, and performs screen display according to a display signal from the control unit 11.

  The storage unit 14 is configured by an HDD (Hard Disk Drive) or the like, and stores various programs and various data.

  The connection unit 15 is configured by a bus interface such as USB (Universal Serial Bus) and inputs image data of moving images output from the cameras 1 to 3, while moving images or images are input to the printer 5 and the media writing unit 6. Output image data of still images.

Returning to FIG. 1, the HUB 4 is a line concentrator interposed between the cameras 1 to 3 and the image processing apparatus 10 and connects them.
The printer 5 performs print processing on a print medium (for example, paper) in accordance with a print instruction from the image processing apparatus 10. The printer 5 is configured by a printing device such as a page printer, for example.
The media writing unit 6 is a device that writes image data of a digital moving image to a medium such as a DVD (portable recording medium) in accordance with a writing instruction from the image processing apparatus 10. The media writing unit 6 may be built in the image processing apparatus 10.

Next, a procedure for displaying or analyzing the trajectory of a moving object using the image processing system 100 configured as described above will be described with reference to FIG.
In the following description, it is assumed that the swing operator at the time of tennis swing is photographed by the cameras 1 to 3 at the same time, and image data of the photographed moving image in each direction is stored by the image processing apparatus 10.

  First, under a certain shooting environment (first shooting environment), the cameras 1 to 3 perform shooting of a swing using a sample form (ideal form) such as a swing of an instructor or a professional player. Note that the number of cameras used for shooting is not limited to three, and may be one or two or more. Image data of a captured moving image (moving image data of an ideal form) is stored in, for example, the storage unit 14 under the control of the control unit 11 of the image processing apparatus 10 (step S101).

  Next, at the shooting position (under the second shooting environment) different from the shooting of the swing with the ideal form, shooting of the swing for which the form is to be checked is performed by the cameras 1 to 3. Here, the camera to shoot may be any of the cameras 1 to 3. Image data of a captured moving image (moving image data to be checked) is stored in, for example, the storage unit 14 under the control of the control unit 11 of the image processing apparatus 10 (step S102).

  Next, the control unit 11 reads out the moving image data of the ideal form from the storage unit 14, and registers various data for acquiring the trajectory of the tennis racket that is the movement target (step S103). Here, with reference to FIG. 5, the procedure of the data registration process for acquiring the locus | trajectory of a tennis racket is demonstrated. In the following description, the contents of registration of various data using moving image data of an ideal form will be described. However, registration of various data using moving image data to be checked is performed in the same procedure. Is called.

First, the control unit 11 specifies a form start frame from the read moving image data of the ideal form (step S201). That is, the control unit 11 designates a frame for starting form diagnosis. More specifically, for example, as shown in FIG. 6, the control unit 11 inputs the selection buttons B1 and B2 displayed on the display screen of the display unit 13 to input the ideal displayed on the screen. The still images constituting the moving image of the form are switched and displayed in time series. FIG. 6 shows a state where the swing operator Q1 holding the tennis racket R1 is swinging on the tennis court. The tennis court is surrounded by a back line BL1 and a side line SL1 that intersect at the corner C1. The selection buttons B1 and B2 displayed on the display screen can be input by operating a mouse as the input unit 12, for example. Then, the control unit 11 designates the form start frame by inputting the set button B3 when an appropriate still image is displayed as the form start frame on the display screen.

Next, the control unit 11 causes the first setting unit 11a to function and designates the backline and the sideline using the still image of the designated form start frame (step S202). Specifically, for example, as illustrated in FIG. 7, the control unit 11 performs a displacement operation on the positions of the pointers P _{1 to} P ₃ displayed on the display screen of the display unit 13, so that the backlines BL 1 and Side line SL1 is defined by lines L ₁ and L ₂ . The pointers P _{1 to} P ₃ displayed on the display screen can be displaced by, for example, a drag operation using the mouse as the input unit 12. The positions of the pointers P _{1 to} P ₃ are stored in, for example, a RAM provided in the control unit 11.

Next, the control unit 11 causes the third setting unit 11i to function and designates the position of the swing operator Q1 (step S203). Specifically, for example, as shown in FIG. 8, the control unit 11 performs a displacement operation on the positions of the pointers P ₄ and P ₅ displayed on the display screen of the display unit 13, so that the swing operator Q 1 define the position by the line L _3. The provisions of the line L ₃ is, for example, can be carried out by defining the neck and part of the waist of the swing operation's Q1 pointer P _4, P _5, respectively. The pointers P ₄ and P ₅ displayed on the display screen can be displaced by, for example, a drag operation with the mouse as the input unit 12.
The positions of the pointers P ₄ and P ₅ are stored in, for example, a RAM provided in the control unit 11.

Next, the control part 11 makes the image designation | designated part 11p function, and designates the position of the tennis racket R1 (step S204). Specifically, for example, as illustrated in FIGS. 9 and 10, when the position of the pointers T ₁ and T ₂ displayed on the display screen of the display unit 13 is displaced, the control unit 11 performs the pointer operation. A rectangular frame FT is defined by the positions of T ₁ and T ₂ . The tennis racket R1 can be specified, for example, by defining the upper left part and the lower right part of the tennis racket R1 with pointers T ₁ and T ₂ as shown in FIG. The pointers T ₁ and T ₂ displayed on the display screen can be displaced by, for example, a drag operation using the mouse as the input unit 12. The positions of the pointers T ₁ and T ₂ are stored in, for example, a RAM provided in the control unit 11.

  Next, the control unit 11 designates a backswing start frame from the moving image data of the ideal form (step S205). That is, the control unit 11 designates a frame corresponding to the start of the backswing operation. More specifically, for example, as shown in FIG. 11, the control unit 11 inputs the selection buttons B1 and B2 displayed on the display screen of the display unit 13 to input the ideal displayed on the screen. The still images constituting the moving image of the form are switched and displayed in time series. And the control part 11 designates a backswing start frame by inputting set button B3 when a suitable still image is displayed as a backswing start frame on a display screen.

  Next, the control unit 11 designates a follow swing start frame from the moving image data of the ideal form (step S206). That is, the control unit 11 designates a frame corresponding to the start of the follow swing operation. More specifically, for example, as shown in FIG. 12, the control unit 11 receives the ideal buttons displayed on the screen when the selection buttons B1 and B2 displayed on the display screen of the display unit 13 are input. The still images constituting the moving image of the form are switched and displayed in time series. Then, the control unit 11 designates the follow swing start frame by inputting the set button B3 when an appropriate still image is displayed as the follow swing start frame on the display screen.

  Next, the control unit 11 designates the follow swing end frame from the moving image data of the ideal form (step S207). That is, the control unit 11 designates a frame corresponding to the end of the follow swing operation. More specifically, for example, as shown in FIG. 13, the control unit 11 inputs the selection buttons B1 and B2 displayed on the display screen of the display unit 13 to input the ideal displayed on the screen. The still images constituting the moving image of the form are switched and displayed in time series. Then, the control unit 11 specifies the follow swing end frame by inputting the set button B3 when an appropriate still image is displayed as the follow swing end frame on the display screen.

  When the registration of the ideal form data is completed as described above, the control unit 11 causes the first imaging angle calculation unit 11e to function as shown in FIG. The shooting angle with respect to the back line of the tennis court (the shooting angle of the ideal form) is calculated (step S104).

Specifically, as illustrated in FIG. 14, the control unit 11 obtains inclinations of the back line and the side line defined by the lines L ₁ and L _{2 with} respect to the horizontal line HL, that is, the angles α and β. Here, the situation of the tennis court when a swing with an ideal form is photographed is shown in a plan view as shown in FIG. Note that a three-dimensional view is shown in FIG. At this time, an axis that connects one camera CM of the cameras 1 to 3 and a corner that is an intersection of the back line BL and the side line SL is defined as a y-axis, and an axis that is in contact with the corner and orthogonal to the y-axis is defined as an axis. Assuming the x axis, the angle formed by the x axis and the back line BL is α ₀ , and the angle formed by the x axis and the side line SL is β ₀ . At this time, the angle formed by the back line BL and the y axis is the shooting angle θ. When the tennis court is viewed in plan, the angle formed between the back line BL and the side line SL is 90 °, and the relationship of the following formula (1) is established. Note that a represents a conversion coefficient.
(A × sin α ₀ ) ² + (a × sin β ₀ ) ² = 1 (1)

Furthermore, when this is expressed on the xyz axis, it is as shown in FIG. In plan view, the viewpoint is on the z axis, and in order to make this the actual viewpoint of the camera CM, the whole is rotated by an angle δ with the x axis as the rotation axis. Then, an arbitrary point X on the xyz axis is rotated on the x axis to become X ′. That is, if the coordinates of the point X are (x, y, z) and the coordinates of the point X ′ are (x ′, y ′, z ′), the coordinates of the point X ′ are as shown in the following equation (2). Can be sought.

According to the above equation (2), it can be seen that the conversion coefficient a is 1 / cos δ.
Then, assuming that the height on the z-axis at the corner position, which is the intersection of the back line BL and the side line SL, is 0, the following expressions (3) and (4) are satisfied.
sin α = cos δ × sin α ₀ (3)
sin β = cos δ × sin β ₀ (4)

That is, the following formula (5) can be expressed from the above formula (3) and the above formula (4).
(1 / cos δ × sin α) ² + (1 / cos δ × sin β) ² = 1 (5)
Therefore, the relationship of the following formula (6) and the following formula (7) is established.
cos θ = 1 / cos δ × sin β (6)
sin θ = 1 / cos δ × sin α (7)
Thereby, the imaging angle θ can be obtained. In the above description, the shooting angle θ for one camera CM among the cameras 1 to 3 is obtained, but the shooting angle θ can be obtained for the other cameras in the same procedure.

When the shooting angle of the ideal form is calculated as described above, the control unit 11 causes the first trajectory acquisition unit 11c to function, and the tennis racket to be moved from the moving image of the swing with the captured ideal form. A trajectory is acquired (step S105). Specifically, the control unit 11 causes the movement target image extraction unit 11r to function and configures a moving image from the follow swing start frame to the follow swing end frame among the swing images of the ideal form. For each of the still images, an image of the tennis racket portion is extracted.
That is, the control unit 11 first causes the image designating unit 11p to function and uses the tennis racket image designated as described above as a template. Next, the control unit 11 selects, for each of the still images constituting the moving image from the follow swing start frame to the follow swing end frame, of the ideal form of the moving image, an image matching the template image. Extract the part. The control unit 11 plots the extracted image as an image of the tennis racket portion, and plots the center of the extracted image as a trajectory. As a result, for example, trajectories H _{1 to} H _n for each frame as shown in FIG. 17 can be acquired. As a template matching method, a publicly known one can be adopted.For example, from a still image of a frame next to a frame in which a template is first set, a fixed range near the coordinates of an image as a template is set as a search range. An area having the smallest difference in parameters of images such as RGB is specified. By repeating the same procedure for the still images of the subsequent frames, the portion of the image that matches the template is specified for each frame.

  The control unit 11 performs the above-described processing on each moving image obtained by shooting with each of the cameras 1 to 3.

  Next, various data for obtaining the trajectory of the tennis racket, the shooting angle, and the frame, which are generated as described above, from the moving image data of the ideal form obtained by each of the cameras 1-3. For example, the data is stored in the storage unit 14 as an ideal form data file together with the trajectory data and the like (step S106). An ideal form data file is designated with a file name. An ideal form data file is generated every time moving image data of an ideal form is created.

Next, the control unit 11 causes the second setting unit 11b, the fourth setting unit 11j, and the like to function so as to read the moving image data to be checked from the storage unit 14 and acquire the trajectory of the tennis racket that is the movement target. The various data are registered (step S107). The data registration processing procedure for acquiring the tennis racket trajectory is the same as the data registration processing procedure for acquiring the tennis racket trajectory for the swing moving image in the ideal form described above. That is, the control unit 11 displays the moving image of the swing to be checked as shown in FIG. 18 on the display screen of the display unit 13, and then specifies the form start frame, the definition of the backline BL2 and the sideline SL2, Designation of the position of the swing operator Q2, designation of the position of the tennis racket R2, designation of the back swing start frame, designation of the follow swing start frame and designation of the follow swing end frame are performed.

  Subsequently, the control unit 11 causes the second shooting angle calculation unit 11f to function, and calculates a shooting angle with respect to the back line of the tennis court when the check subject's swing is shot (the shooting angle of the check target person). (Step S108). Calculation of the shooting angle of the person to be checked can be performed by the same procedure as the calculation of the shooting angle of the ideal form described above.

  Next, the control unit 11 causes the second trajectory acquisition unit 11d to function, and acquires the trajectory of the tennis racket that is the movement target from the captured moving image of the person to be checked (step S109). The method of acquiring the trajectory of the tennis racket in the moving image of the swing of the person to be checked can be performed by the same procedure as the method of acquiring the trajectory of the tennis racket in the moving image of the swing using the ideal form described above.

  Next, the control unit 11 acquires the trajectory of the tennis racket generated as described above from the moving image data to be checked obtained by shooting with one or more of the cameras 1 to 3. Are stored in the storage unit 14, for example, as a form data file to be checked (step S110). Note that the file name is specified for the form data file to be checked.

  Thereafter, the control unit 11 reads the form data file to be checked from the storage unit 14 (step S111), and then reads the ideal form data file from the storage unit 14 (step S112). When reading the ideal form data file, the control unit 11 displays a file selection screen as shown in FIG. 19 on the display screen of the display unit 13, for example. For example, the user can select any ideal form data file by operating the mouse or keyboard as the input unit 12 and selecting one from the list on the file selection screen.

Next, the control unit 11 causes the correction unit 11h, the ratio calculation unit 11k, and the position difference calculation unit 11m to function, and the trajectory of the tennis racket regarding the moving image of the swing with the ideal form included in the file read in step S112. The data position is corrected (step S113).
Here, in this embodiment, the cameras 1 to 3 shoot a swing with an ideal form at three shooting angles, and acquire respective moving images. Therefore, a moving image that is a target for correcting the position of the track data of the tennis racket is selected. The selection of a moving image that is to be corrected for the position of the track data of the tennis racket is based on the moving image obtained by shooting the swing to be checked for each of the shooting angles of the moving image obtained by shooting the swing with an ideal form. This is performed by specifying a moving image having the smallest difference in shooting angle compared to the shooting angle.

Specifically, as shown in FIG. 20 (A), first, the tennis racket is viewed from the center of the body of the swing operator when the situation of the tennis court when the swing with the ideal form is photographed is shown in plan view. The distance between the center of the swinging operator and the center RC of the tennis racket with respect to the direction of the hitting ball, which is the direction in which the tennis ball flies (for example, the direction of the cross) Can be represented by an angle ε. At this time, the angle formed by the direction of the hit ball with respect to the back line BL is, for example, 60 °. At this time, the angle of the line connecting the center of the swing operator's body and the center RC of the tennis racket with respect to a line orthogonal to the axis that forms the shooting angle θ with respect to the backline BL can be represented by an angle γ. Then, assuming that the conversion coefficient of the moving image of the swing with the ideal form with respect to the moving image of the swing to be checked is A, the camera CM that forms the shooting angle θ with respect to the back line BL in the moving image of the swing with the ideal form. A distance X (see FIGS. 20A and 20B) between the axis passing through and the center RC of the tennis racket can be expressed by the following equation (8).
X = A × r × cos γ (8)
For example, as shown in FIG. 21, the conversion coefficient A is the length of the line L ₄ defined in the moving image of the swing to be checked with respect to the length of the line L ₃ defined in the moving image of the swing with an ideal form. It can be determined by the ratio of the length.

Here, the angle γ can be expressed by the following equation (9).
γ = abs (90 ° − (ε−60 ° + θ)
= Abs (150 ° −ε−θ) (9)

When the amount of deviation between the shooting angle in the moving image of the swing in the ideal form and the shooting angle in the moving image of the check target swing is Δθ, the distance X ′ when the shooting angle in the moving image of the check target swing is used. Can be represented by the following formula (10). That is, the distance X can be converted into the distance X ′. The photographing angle deviation amount Δθ is obtained by the control unit 11 having the photographing angle difference calculation unit 11g function.
X ′ = A × r × cos (γ + Δθ)
= A × r × cos γ × cos Δθ−A × r × sin γ × sin Δθ
= X × cos Δθ−A × r × sin γ × sin Δθ (10)

For example, when the deviation amount Δθ between the shooting angle in the moving image of the swing with the ideal form and the shooting angle in the moving image of the swing to be checked is 10 °, X ′ is obtained by the following equation (11).
X ′ = 0.98 × X−A × r × sin γ × 0.17 (11)

Here, the above-mentioned angle ε in the ideal form is obtained. The angle ε is set, for example, in advance in a swing moving image of an ideal form by setting a time for a back swing start state, a time for a follow swing start state, and a time for a follow swing end state, respectively. Therefore, it can be predicted over time. For example, if the angle ε at the start of the backswing is 0 °, the angle ε at the end of the backswing, that is, the start of the follow swing is 180 °, and the angle ε at the end of the follow swing Becomes 0 °. These times can be specified by, for example, the backswing start frame, follow swing start frame, and follow swing end frame specified as described above.

When the angle ε is obtained, the angle γ can be obtained from the above equation (9). As a result, the solution of “A × r × sin γ” in the above equation (10) is obtained. Can be sought.
In this way, the trajectory H _{1 to} H _n for each frame of the ideal form can be converted from the distance X to the distance X ′.

Then, the control unit 11 shifts the position of the ideal form trajectory converted as described above so as to correspond to the position of the swing operator Q2 in the moving image of the swing to be checked. Thereby, it becomes possible to perform comparison display with the trajectory of the tennis racket of the check subject in the moving image of the swing to be checked. Specifically, the control unit 11 coordinates (X ₁ , Y ₁ ) to (X _n , Y _n ) of the trajectories H _{1 to} H _n for each frame of the ideal form converted as described above. Are respectively shifted and corrected to (X ₁ ′, Y ₁ ′) to (X _n ′, Y _n ′). As shown in FIG. 21, the corrected coordinates of the trajectories H _{1 to} H _n are coordinates P ₄ (x0) of the pointers P ₄ and P ₅ defined as described above by the function of the third setting unit 11i. , Y0), P ₅ (x1, y1) and the coordinates P ₆ (x2, y2), P ₇ (x3) of the pointers P ₆ and P ₇ defined as described above by the function of the fourth setting unit 11j. , Y3
) And the following equation (12) and the following equation (13). In the following formulas (12) and (13), the conversion formula from the coordinates (X _n , Y _n ) of the trajectory H _n for each frame of the ideal form to the coordinates (X _n ′, Y _n ′). The coordinates of other trajectories can be obtained in the same manner.
_Xn '= (y0-y1) / (y2-y3) * _Xn + (x2-x0) ... (12)
_{Y n '= (y0-y1} ) / (y2-y3) × Y n + (y2-y0) ··· (13)

After correcting the position of the trajectory of the tennis racket for the moving image of the swing with the ideal form as described above, the control unit 11 performs the trajectory of the tennis racket of the corrected ideal form as shown in FIG. And display data for displaying the trajectory of the tennis racket of the person to be checked on the display unit 13 (step S114). The trajectory for each frame may be displayed as a point, or may be displayed as a line connecting the points. At this time, the timing of the swing of the tennis racket may be different between the swing moving image of the ideal form and the moving image of the swing to be checked. For example, as shown in FIG. It is preferable to match the playback timing of the backswing start frame designated as described above. Then, as shown in FIG. 22A, when the swing speed differs between the swing of the ideal form and the swing of the person to be checked, this can be displayed so that the user can recognize it. . In this case, when it is desired to match the swing speed between the swing of the ideal form and the swing of the person to be checked, for example, as shown in FIG. Add still images to increase. The addition of the still image can be realized, for example, by inserting the same still image as the still image of a part of the frame of the swing moving image in an ideal form. Further, still images of some frames of the moving image of the swing to be checked may be thinned out. In this way, the difference between the trajectory of the ideal form tennis racket and the trajectory of the tennis racket of the person to be checked can be understood more clearly. And the control part 11 diagnoses the locus | trajectory of a tennis racket of a check subject by operating the locus | trajectory difference determination part 11n, and determining the difference of these locus | trajectories (step S115). Specifically, for example, the control unit 11 determines whether or not the trajectory difference at the start of the follow swing, the trajectory difference at the end of the follow swing, and the like have a predetermined size. And the control part 11 produces | generates the display data for displaying the message corresponding to a diagnostic result (step S116). The control unit 11 displays a moving image of the swing to be checked on the display screen of the display unit 13 and performs display based on the display data generated in Step S114 and Step S116 (Step S117). As a result, for example, a screen as shown in FIG. 23 is displayed on the display screen of the display unit 13. In FIG. 23, the difference between the corrected ideal form tennis racket trajectory and the check target tennis racket trajectory at the end of the follow swing exceeds a predetermined size, so the corrected ideal form Along with the track SW1 of the tennis racket and the track SW2 of the tennis racket of the person to be checked, for example, “Let's make follow-through higher” is displayed. In addition, you may make it display each locus | trajectory sequentially according to the moving image of the displayed flame | frame. For example, at the timing when the moving image of the 30th frame is displayed on the display screen of the display unit 13, the locus from the 1st to the 30th frame is displayed, and the locus after the 31st frame is not displayed. Further, in the present embodiment, when a moving image of a swing to be checked is displayed, a still image at a representative timing such as the moment of hitting a hit ball or the timing of starting a swing can be displayed. At this time, the trajectory of the corrected ideal form tennis racket corresponding to the displayed still image is displayed.

  Next, as another example of the present embodiment, a case will be described in which the trajectory of a golf swing at a driving range is displayed or analyzed using the image processing system 100 described above. Note that each step shown in the following description corresponds to each process in FIGS. 4 and 5, so these drawings will be referred to.

First, under a certain shooting environment (first shooting environment), as described above, shooting of a swing with an ideal form is performed by the cameras 1 to 3. The captured moving image data of the ideal form is stored in, for example, the storage unit 14 under the control of the control unit 11 of the image processing apparatus 10 (step S101).
In addition, at the shooting position (under the second shooting environment) different from the shooting of the swing with the ideal form, as described above, shooting of the swing for which the form is to be checked is performed by the cameras 1 to 3. The captured moving image data to be checked is stored in, for example, the storage unit 14 under the control of the control unit 11 of the image processing apparatus 10 (step S102).

  Next, the control unit 11 reads out the moving image data of the ideal form from the storage unit 14, and registers various data for acquiring the trajectory of the golf club that is the movement target (step S103).

Here, the procedure of the data registration process for acquiring the trajectory of the golf club will be described. First, the control unit 11 designates the form start frame from the read moving image data of the ideal form (step S201). . More specifically, for example, as shown in FIG. 24, the control unit 11 inputs the selection buttons B1 and B2 displayed on the display screen of the display unit 13 to input the ideal displayed on the screen. The still images constituting the moving image of the swing by the form are switched and displayed in time series. FIG. 24 shows a state where the swing operator Q3 holding the golf club W1 is swinging on the mat M1. The mat M1 is surrounded by the left end portion ML1, the tip portion MT1, and the right end portion MR1. Then, the control unit 11 designates the form start frame by inputting the set button B3 when an appropriate still image is displayed as the form start frame on the display screen.

Next, the control unit 11 causes the first setting unit 11a to function and designates the left end portion ML1, the tip end portion MT1, and the right end portion MR1 of the mat M1 using the still image of the designated form start frame (step) S202). Specifically, for example, as shown in FIG. 25, the control unit 11 moves the positions of the pointers P _{11 to} P ₁₄ displayed on the display screen of the display unit 13 to move the left end of the mat. ML1, the tip portion MT1, and the right end portion MR1 are defined by lines L _{11 to} L ₁₃ , respectively. In the present embodiment, the three points of the left end ML1, the leading end MT1 and the right end MR1 of the mat are defined to improve the accuracy of calculation of the photographing angle, but two of these are defined. Then, the photographing angle may be calculated. The positions of the pointers P _{11 to} P ₁₄ are stored in, for example, a RAM provided in the control unit 11.

Next, the control unit 11 causes the third setting unit 11i to function and designates the position of the swing operator Q3 (step S203). Specifically, for example, as shown in FIG. 26, the control unit 11 operates the swing operator Q3 by moving the positions of the pointers P ₁₅ and P ₁₆ displayed on the display screen of the display unit 13. define the position by the line _{L 14.} The positions of the pointers P ₁₅ and P ₁₆ are stored in, for example, a RAM provided in the control unit 11.

Next, the control unit 11 causes the image specifying unit 11p to function and specifies the position of the golf club W1 (step S204). Specifically, for example, as shown in FIGS. 27 and 28, when the positions of the pointers T ₁₁ and T ₁₂ displayed on the display screen of the display unit 13 are displaced, the control unit 11 performs the pointer operation. A rectangular frame FG is defined by the positions of T ₁₁ and T ₁₂ . For example, as shown in FIG. 28, the golf club W1 can be defined by defining the upper left portion and the lower right portion of the head of the golf club W1 with pointers T ₁₁ and T ₁₂ , respectively. The positions of the pointers T ₁₁ and T ₁₂ are stored in, for example, a RAM provided in the control unit 11.

  Next, the control unit 11 designates a backswing start frame from the moving image data of the ideal form (step S205). More specifically, for example, as shown in FIG. 29, the control unit 11 receives the selection buttons B1 and B2 displayed on the display screen of the display unit 13 to input the ideal displayed on the screen. The still images constituting the moving image of the swing by the form are switched and displayed in time series. And the control part 11 designates a backswing start frame by inputting set button B3 when a suitable still image is displayed as a backswing start frame on a display screen.

  Next, the control unit 11 designates a follow swing start frame from the moving image data of the ideal form (step S206). More specifically, for example, as shown in FIG. 30, the control unit 11 receives the selection buttons B1 and B2 displayed on the display screen of the display unit 13 to input the ideal displayed on the screen. The still images constituting the moving image of the swing by the form are switched and displayed in time series. Then, the control unit 11 designates the follow swing start frame by inputting the set button B3 when an appropriate still image is displayed as the follow swing start frame on the display screen.

  Next, the control unit 11 designates the follow swing end frame from the moving image data of the ideal form (step S207). More specifically, for example, as shown in FIG. 31, the control unit 11 receives the ideal buttons displayed on the screen when the selection buttons B1 and B2 displayed on the display screen of the display unit 13 are input. The still images constituting the moving image of the swing by the form are switched and displayed in time series. Then, the control unit 11 specifies the follow swing end frame by inputting the set button B3 when an appropriate still image is displayed as the follow swing end frame on the display screen.

  When the registration of the ideal form data is completed as described above, the control unit 11 causes the first shooting angle calculation unit 11e to function and calculates the shooting angle of the ideal form (step S104).

Specifically, as shown in FIG. 32, the control unit 11 tilts the left end portion ML1, the tip end portion MT1, and the right end portion MR1 of the mat M1 defined by the lines L _{11 to} L _{13 with} respect to the horizontal line HL, that is, the angle. Find α, ζ, and β. Here, when the state of the mat when a swing with an ideal form is photographed is represented in plan view, it is as shown in FIG. At this time, an axis connecting the camera CM of one of the cameras 1 to 3 and a corner that is the intersection of the leading end MT and the right end MR of the mat is defined as the y axis, and an axis orthogonal to the y axis is defined as the x axis. Then, the angle formed between the x axis and the right end portion MR of the mat is β ₀ , and the angle formed between the x axis and the front end portion MT of the mat is ζ ₀ .

Here, first, attention is paid to the intersection between the leading end MT of the mat and the right end MR, and the angle θ ₁ formed by the axis connecting the camera CM to the right end MR of the mat and this intersection is obtained. When the mat is viewed in a plan view, the angle formed between the leading end MT and the right end MR of the mat is 90 °, and according to FIG. 33, it can be seen that the angle θ ₁ matches the angle ζ _0. The relationship of Formula (14) is established. Incidentally, _{a 1} represents a conversion coefficient.
(A ₁ × sin β ₀ ) ² + (a ₁ × sin ζ ₀ ) ² = 1 (14)

As described above, when this is represented on the xyz axis, the viewpoint is on the z axis in the plan view. Therefore, in order to make this the actual camera CM viewpoint, the x axis is the rotation axis. As a whole, the angle δ is rotated _once . Then, according to the above, it can be seen that the conversion coefficient a ₁ is 1 / cos δ ₁ .
Then, assuming that the height on the z-axis at the corner position, which is the intersection of the tip portion MT and the right end portion MR of the mat, is 0, the following equations (15) and (16) are satisfied.
sin β = cos δ ₁ × sin β ₀ (15)
sinζ = cosδ ₁ × sinζ ₀ (16)

That is, the following formula (17) can be expressed from the above formula (15) and the above formula (16).
(1 / cos δ ₁ × sin β) ² + (1 / cos δ ₁ × sin ζ) ² = 1 (17)
Therefore, the relationship of the following formula (18) and the following formula (19) is established.
cos θ ₁ = 1 / cos δ ₁ × sin β (18)
sin θ ₁ = 1 / cos δ ₁ × sin ζ (19)
Thereby, angle (theta) ₁ can be calculated | required.

Next, with reference to FIG. 34, it noted the intersection of the mat tip MT and left portion ML, determined for the angle theta ₂ of the axis connecting the intersection point with the camera CM for mat left end ML. When the mat is viewed in plan, the angle formed between the leading end MT and the left end ML of the mat is 90 °, and the relationship of the following formula (20) is established. Incidentally, _{a 2} indicates the conversion factor.
(A ₂ × sin α ₁ ) ² + (a ₂ × sin ζ ₁ ) ² = 1 (20)

As described above, when this is represented on the xyz axis, the viewpoint is on the z axis in the plan view. Therefore, in order to make this the actual camera CM viewpoint, the x axis is the rotation axis. As a whole, the angle δ is rotated _twice . Then, according to the above, it can be seen that the conversion coefficient a ₂ is 1 / cos δ ₂ .
Then, assuming that the height on the z-axis of the corner position, which is the intersection of the front end MT and the left end ML of the mat, is 0, the following relations (21) and (22) are established.
sin α = cos δ ₂ × sin α ₁ (21)
sinζ = cosδ ₂ × sinζ ₁ (22)

That is, the following formula (23) can be expressed from the above formula (21) and the above formula (22).
^{_{(1 / cosδ 2 × sinα)}} 2 + (1 / cosδ 2 × sinζ) 2 = 1 ··· (23)
Therefore, the relationship of the following formula (24) and the following formula (25) is established.
cos θ ₂ = 1 / cos δ ₂ × sin α (24)
sin θ ₂ = 1 / cos δ ₂ × sin ζ (25)
Thereby, angle (theta) ₂ can be calculated | required.

Next, the shooting angle θ is obtained from the angle θ ₁ and the angle θ ₂ .
As shown in FIG. 35, when the mat is viewed in plan, the shooting angle θ can be expressed by the following equation (26).
θ = h ₂ ′ / (h ₁ ′ + h ₂ ′) × θ ₁ + h ₁ ′ / (h ₁ ′ + h ₂ ′) × θ ₂ (26)

Relationship shown in the equation (26), since substantially the same even on the display screen, as shown in FIG. 36, the horizontal between the center WC of intersection and the golf club head of the line L ₁₁ and the line L ₁₂ When the horizontal distance h ₂ between the direction distance h ₁ and the intersection of the line L ₁₂ and the line L ₁₃ and the center WC of the golf club head is obtained, the shooting angle θ is calculated by the following equation (27). Can do. The center WC of the golf club head is the center of the frame FG defined by the positions of the pointers T ₁₁ and T ₁₂ as described above.
θ = h ₂ / (h ₁ + h ₂ ) × θ ₁ + h ₁ / (h ₁ + h ₂ ) × θ ₂ (27)
Thereby, the imaging angle θ can be obtained. In the above description, the shooting angle θ for one camera CM among the cameras 1 to 3 is obtained, but the shooting angle θ can be obtained for the other cameras in the same procedure.

When the shooting angle of the ideal form is calculated as described above, the control unit 11 causes the first trajectory acquisition unit 11c to function, and the golf club that is the movement target from the moving image of the swing with the captured ideal form. A trajectory is acquired (step S105). Specifically, the control unit 11 causes the movement target image extraction unit 11r to function and configures a moving image from the follow swing start frame to the follow swing end frame among the swing images of the ideal form. For each still image, an image of the golf club portion is extracted. That is, first, the control unit 11 causes the image designating unit 11p to function and uses the golf club head image designated as described above as a template. Next, the control unit 11 matches each of the still images constituting the moving image from the follow swing start frame to the follow swing end frame with the image used as the template among the swing moving images based on the ideal form. Extract a part of an image. The control unit 11 plots the extracted image as an image of the head portion of the golf club, and plots the center of the extracted image as a trajectory. As a result, the trajectory for each frame can be acquired as described above.

  The control unit 11 performs the above-described processing on each moving image obtained by shooting with each of the cameras 1 to 3.

  Next, various data for obtaining the trajectory of the golf club, the shooting angle, and the frame generated as described above from the moving image data of the ideal form obtained by each of the cameras 1 to 3. For example, the data is stored in the storage unit 14 as an ideal form data file together with the trajectory data and the like (step S106). An ideal form data file is designated with a file name. An ideal form data file is generated every time moving image data of an ideal form is created.

  Next, the control unit 11 causes the second setting unit 11b, the fourth setting unit 11j, and the like to function, reads moving image data to be checked from the storage unit 14, and acquires the trajectory of the golf club that is the moving target. The various data are registered (step S107). The data registration processing procedure for acquiring the golf club trajectory is the same as the data registration processing procedure for acquiring the golf club trajectory regarding the swing moving image in the ideal form described above.

  Subsequently, the control unit 11 causes the second shooting angle calculation unit 11f to function and calculates a shooting angle (a shooting angle of the check target person) when the swing of the check target is shot (step S108). Calculation of the shooting angle of the person to be checked can be performed by the same procedure as the calculation of the shooting angle of the ideal form described above.

  Next, the control unit 11 causes the second trajectory acquisition unit 11d to function, and acquires the trajectory of the golf club that is the movement target from the captured moving image of the person to be checked (step S109). The method of acquiring the golf club locus in the swing image of the person to be checked can be performed by the same procedure as the method of acquiring the golf club locus in the swing image of the ideal form described above.

  Next, the control unit 11 acquires the golf club trajectory generated as described above from the moving image data to be checked obtained by shooting with one or more of the cameras 1 to 3. Are stored in the storage unit 14, for example, as a form data file to be checked (step S110). Note that the file name is specified for the form data file to be checked.

  Thereafter, the control unit 11 reads the form data file to be checked from the storage unit 14 (step S111), and then reads the ideal form data file from the storage unit 14 (step S112). The procedure for reading an ideal form data file is as described above.

  Next, the control unit 11 causes the correction unit 11h, the ratio calculation unit 11k, and the position difference calculation unit 11m to function, and the golf club trajectory regarding the swing moving image by the ideal form included in the file read in step S112. The data position is corrected (step S113). The procedure for selecting a moving image, which is a target for correcting the position of the golf club trajectory data, is as described above.

Specifically, first, as shown in FIG. 37, when the state of the mat of the golf driving range when a swing with an ideal form is photographed is shown in plan view, the golf club starts from the center of the body of the swing operator. The distance to the center WC of the head of the golf club is represented by a distance r, and the angle of the line connecting the center of the swinging operator's body and the center WC of the golf club with respect to the direction perpendicular to the direction of the hit ball is represented by an angle ε. it can. At this time, the angle of the line connecting the center of the swinging operator's body and the center WC of the golf club head with respect to a line orthogonal to the axis forming the shooting angle θ with respect to the right end portion MR of the mat is represented by an angle γ. Can do. Then, in the moving image of the swing of the person to be checked, the distance X between the axis passing through the camera CM formed by the shooting angle θ with respect to the right end MR of the mat and the center WC of the golf club head (see FIGS. 37 and 38). Is the above formula (8)
Can be expressed similarly.
Note that, as described above, the conversion coefficient A is obtained by the ratio of the line length defined in the moving image of the swing of the person to be checked to the length of the line defined in the moving image of the swing with an ideal form. it can.

Here, the angle γ can be expressed by the following equation (28).
γ = ε−θ (28)

  The distance when the shooting angle in the moving image of the swing of the check target is Δθ is the distance between the shooting angle in the moving image of the swing of the ideal form and the shooting angle in the moving image of the swing of the check target. X ′ can be similarly expressed by the above formula (10). The photographing angle deviation amount Δθ is obtained by the control unit 11 having the photographing angle difference calculation unit 11g function.

  Again, the above-mentioned angle ε in the ideal form is obtained in the manner described above. For example, when the angle ε at the start of the back swing is 0 °, the angle ε at the start of the follow swing is 180 °. Further, the angle ε at the time of impact is 0 °, and the angle ε at the end of the follow swing is 180 °.

Then, when the angle ε is obtained, the angle γ can be obtained from the above equation (28), and as a result, a solution of “A × r × sin γ” in the above equation (10) is obtained. Can be sought.
In this way, the trajectory for each frame of the ideal form can be converted from the distance X to X ′.

  Then, the control unit 11 shifts the position of the ideal form trajectory converted as described above so as to correspond to the position of the swing operator in the moving image of the check target person's swing. Thereby, it becomes possible to perform comparison display with the trajectory of the golf club of the check target in the moving image of the swing of the check target. The procedure for shifting the position of the ideal form trajectory can be performed in the same manner as described above.

After correcting the position of the golf club locus for the swing image of the ideal form as described above, the control unit 11, as shown in FIG. 4, corrects the golf club locus of the ideal form. And the display data for displaying on the display part 13 the locus | trajectory of a golf club of a check subject are produced | generated (step S114). The trajectory for each frame may be displayed as a point, or may be displayed as a line connecting the points. And the control part 11 makes the locus | trajectory difference determination part 11n function as needed, diagnoses the locus | trajectory of a golf club of a check subject (step S115), and displays the message corresponding to a diagnostic result. Display data is generated (step S116). The control unit 11 displays the moving image to be checked on the display screen of the display unit 13 and performs display based on the display data generated in step S114 and step S116 (step S117). As a result, for example, a screen as shown in FIG. 39 is displayed on the display screen of the display unit 13. In FIG. 39, the corrected ideal form golf club locus SW3 and the check target golf club locus SW4 are displayed.

  In the above-described embodiment, the center of the image of the golf club head is plotted as a trajectory. However, the line from the center of the swing operator's body to the head of the golf club is set, and this setting is made. You can also plot line trajectories.

Specifically, for example, when the position of the golf club W1 is specified as described above, as shown in FIGS. 40 and 41, the position of the golf club W1 is specified and the swing start point of the golf club W1 is specified. Specify the position of the shoulder of the swing operator. More specifically, for example, when the position of the pointers T _{21 to} T ₂₄ displayed on the display screen of the display unit 13 is displaced, the control unit 11 changes the position according to the positions of the pointers T ₂₁ and T _22. A frame FGa having a shape is defined, and a square frame FSa is defined by the positions of the pointers T ₂₃ and T ₂₄ . As shown in FIG. 41, when the head of the golf club W1 is surrounded by the frame FGa and the shoulder of the swing operator is surrounded by the frame FSa, the centers of the frame FGa and the frame FSa are connected as shown in FIG. A straight line is defined. Then, in the manner described above, the control unit 11 causes the movement target image extraction unit 11r to function, and for each of the still images constituting the moving image from the follow swing start frame to the follow swing end frame, the frame FGa and the frame Using the images defined by FSa as templates, a portion of an image that matches the template image is extracted. Then, the centers of the extracted images are connected by straight lines, and this is used as a locus. For example, as shown in FIG. 43, line-like trajectories J _{1 to} J _n can be acquired by connecting the centers of images matched to the template for each frame with straight lines. Then, the position of the linear trajectories J _{1 to} J _n obtained in this way is corrected as described above, and the result is displayed on the display screen of the display unit 13. As a result, the display of the golf club track can be varied.

As described above, in the present embodiment, the control unit 11 is different from the first feature point in the moving image obtained by photographing in the first photographing environment and the first photographing environment. Based on the difference from the second feature point in the moving image obtained in the shooting environment, the first moving object obtained from the moving image obtained in the first shooting environment The first trajectory is corrected. The display control unit 11s displays the locus corrected by the control unit 11. As a result, even if the shooting environment such as the shooting angle is different between the first shooting environment and the second shooting environment, the position of the first locus is corrected and displayed in accordance with the shooting angle in the second shooting environment. Therefore, the position of the first locus in the second shooting environment can be accurately grasped. Therefore, for example, in order to accurately grasp the position of the first trajectory, there is no need to capture a moving image again in the second imaging environment.

Further, in the present embodiment, the first setting unit 11a sets two or more straight lines that intersect as the first feature points in the moving image obtained by shooting in the first shooting environment with the cameras 1 to 3. To do. The 1st locus | trajectory acquisition part 11c acquires the locus | trajectory (1st locus | trajectory) of the 1st movement object from the moving image obtained by image | photographing on the 1st imaging environment with the cameras 1-3. The second setting unit 11b sets two or more straight lines that intersect as the second feature points in the moving image obtained by shooting with the cameras 1 to 3 in a second shooting environment different from the first shooting environment. To do. The first shooting angle calculation unit 11e calculates the shooting angles of the cameras 1 to 3 when shooting is performed in the first shooting environment from angles formed by two or more straight lines set by the first setting unit 11a. . The second shooting angle calculation unit 11f calculates the shooting angles of the cameras 1 to 3 when shooting is performed in the second shooting environment from the angles formed by the two or more straight lines set by the second setting unit 11b. . The shooting angle difference calculation unit 11g calculates a difference between the shooting angle calculated by the first shooting angle calculation unit 11e and the shooting angle calculated by the second shooting angle calculation unit 11f. The correction unit 11h corrects the position of the first trajectory acquired by the first trajectory acquisition unit 11c based on the difference in the shooting angle calculated by the shooting angle difference calculation unit 11g. The display control unit 11s displays the first locus corrected by the correction unit 11h. As a result, even if the shooting angle differs between the first shooting environment and the second shooting environment, the position of the first trajectory can be corrected and displayed in accordance with the shooting angle in the second shooting environment. It is possible to accurately grasp the position of the first locus in the second imaging environment.

  In the present embodiment, the image designation unit 11p designates the first movement target image. The movement target image extraction unit 11r extracts an image corresponding to the image designated by the image designation unit 11p from the moving images obtained by photographing in the first photographing environment with the cameras 1 to 3. The first trajectory acquisition unit 11c acquires the first trajectory based on the image extracted by the movement target image extraction unit 11r. As a result, the first trajectory can be easily extracted.

Moreover, in this Embodiment, the 3rd setting part 11i sets the straight line as a 3rd feature point in the moving image obtained by image | photographing on the 1st imaging environment with the cameras 1-3. The fourth setting unit 11j sets a straight line as the fourth feature point in the moving image obtained by shooting with the cameras 1 to 3 in the second shooting environment. The ratio calculation unit 11k calculates the ratio of the length of the straight line set by the third setting unit 11i and the straight line set by the fourth setting unit 11j. The correcting unit 11h further corrects the position of the first trajectory acquired by the first trajectory acquiring unit 11c based on the ratio calculated by the ratio calculating unit 11k. As a result, the position of the first trajectory can be displayed, for example, in accordance with the scale of the moving image obtained by shooting in the second shooting environment, so the first trajectory in the second shooting environment. Can be grasped more accurately.

  In the present embodiment, the position difference calculation unit 11m calculates the position difference between the straight line set by the third setting unit 11i and the straight line set by the fourth setting unit 11j. The correction unit 11h further corrects the position of the first trajectory acquired by the first trajectory acquisition unit 11c based on the position difference calculated by the position difference calculation unit 11m. As a result, the position of the first trajectory can be displayed, for example, in accordance with a predetermined position of a moving image obtained by photographing in the second photographing environment, so that the second locus in the second photographing environment can be displayed. The position of one locus can be grasped more accurately.

Moreover, in this Embodiment, the 1st setting part 11a respond | corresponds to two or more straight lines corresponding to the moving image each image | photographed with the some imaging | photography angle by the cameras 1-3 in the 1st imaging | photography environment. Set each. The 1st locus | trajectory acquisition part 11c acquires a 1st locus | trajectory for every imaging | photography. The first shooting angle calculation unit 11e calculates a plurality of shooting angles. The shooting angle difference calculation unit 11g selects and selects the one closest to the shooting angle calculated by the second shooting angle calculation unit 11f from the plurality of shooting angles calculated by the first shooting angle calculation unit 11e. The difference between the shooting angle and the shooting angle calculated by the second shooting angle calculation unit 11f is calculated. The correction unit 11h is a photographic angle selected by the photographic angle difference calculation unit 11g from a plurality of first trajectories acquired by the first trajectory acquisition unit 11c based on the difference in photographic angle calculated by the photographic angle difference calculation unit 11g. Correct the position of the object corresponding to the angle. As a result, the correction amount of the position of the first locus can be reduced, and the accuracy of the correction of the position of the first locus can be increased.

  Moreover, in this Embodiment, the 1st locus | trajectory acquisition part 11c sets the straight line as a 1st movement object, and acquires a locus | trajectory of a straight line as a 1st locus | trajectory. As a result, more various trajectory displays can be performed. For example, more various forms can be analyzed.

Further, in the present embodiment, the second trajectory acquisition unit 11d acquires the trajectory of the second movement target as the second trajectory from the moving image obtained by photographing with the cameras 1 to 3 in the second photographing environment. To do.
The display control unit 11s displays the second trajectory acquired by the second trajectory acquisition unit 11d together with the first trajectory corrected by the control unit 11. As a result, the first trajectory and the second trajectory can be compared and displayed.

  In the present embodiment, the trajectory difference determination unit 11n determines the difference between the corrected first trajectory and the second trajectory acquired by the second trajectory acquisition unit 11d. The display control unit 11s performs display according to the determination result by the trajectory difference determination unit 11n. As a result, for example, it is possible to show appropriate advice about the form specified by the trajectory, and the convenience is improved.

  Further, in the present embodiment, the display control unit 11 s is corrected to correspond to a still image at a predetermined timing among moving images obtained by shooting in the first shooting environment with the cameras 1 to 3. The first trajectory is displayed. As a result, for example, the analysis of the trajectory at a certain timing such as the moment when the hit ball hits becomes easy.

  In the present embodiment, the tennis court line and the golf driving range mat are defined as the first feature point and the second feature point, and the shooting angle is thereby obtained. The point and the second feature point can be set as appropriate according to the subject to be photographed. For example, when shooting a swing of a baseball bat, a batter box can be defined as a feature point. The feature points are not limited to tennis court lines, golf driving range mats, and the like, but may be other things, for example, trees or buildings. At this time, it is preferable to use a fixed object as a target of the feature point.

  In this embodiment, the length of the swing operator's body is defined as the third feature point and the fourth feature point. However, the third feature point and the fourth feature point are also used. It can be set appropriately according to the object to be photographed. For example, as the third feature point and the fourth feature point, the length of the foot of the swing operator, the length from the tip of the head to the toe, or the swinging arm and the tennis racket or golf club to be moved The previous length may be used.

  In the present embodiment, the length of the swing operator's body as the third feature point and the fourth feature point is defined by the pointer. For example, the subject tracking is performed by the template matching method described above. You may make it prescribe | regulate by performing. Specifically, for example, the image around the shoulder of the swing operator and the image around the waist are set as templates, and for each of the still images constituting the moving image from the follow swing start frame to the follow swing end frame, The part of the image that matches the template image is extracted. Then, a line connecting the center of the extracted image around the shoulder and the center of the image around the waist is defined as the body length of the swing operator. Thereby, even when the swing operator moves, the position can be accurately grasped.

  Further, in the present embodiment, various moving image data to be checked obtained by shooting in the second shooting environment are saved, the saved moving image data is read out, and various trajectories for acquiring the moving locus are obtained. Although data registration is performed, various data may be registered using a real-time moving image obtained by shooting in the second shooting environment.

  Further, in the present embodiment, registration of various data for acquiring a movement target locus by a user input using moving image data of an ideal form or moving image data to be checked is performed. Alternatively, it may be capable of performing image recognition and automatically registering various data.

  In the present embodiment, the moving target trajectory of the ideal form and the moving target trajectory of the check target are compared and displayed on the display unit 13 in the swing image of the check target person. Only the trajectory of the movement target of the ideal form may be displayed on the moving image of the subject's swing.

  In the present embodiment, the swing of an ideal form is photographed in the first photographing environment, and the swing of the person to be checked is photographed in the second photographing environment, but the check target is captured in the first photographing environment. A person's swing may be photographed, and a swing with an ideal form may be photographed in the second photographing environment.

  In this embodiment, the trajectory for each frame is acquired by extracting an image that matches the template image. However, the trajectory is manually specified for each frame by operating the input unit 12 or the like. Also good.

  Further, in the present embodiment, the cameras 1 to 3 are used to photograph a swing with an ideal form in the first photographing environment, and the trajectory is acquired using the captured moving image of the swing with the ideal form. However, the image data of the moving image of the swing with the ideal form may not be obtained by photographing with the cameras 1 to 3 but may be obtained by media or communication means.

  Further, in this embodiment, a plurality of types of image data of a moving image of a swing with an ideal form are acquired, and a trajectory is displayed by selection by a user. However, a moving image of a swing with an ideal form to be acquired is used. The image data may be only one type.

  In the present embodiment, the position correction of the trajectory is performed by the ratio calculation unit 11k and the position difference calculation unit 11m, but the position correction by each of these function units may not be performed.

Although the embodiments and modifications of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments and modifications, and the scope of the invention described in the claims and equivalents thereof are described. Includes range.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
A first feature point in a moving image obtained by photographing under a first photographing environment and a second feature in a moving image obtained by photographing under a second photographing environment different from the first photographing environment. A trajectory correcting means for correcting a trajectory that is a moving object obtained from a moving image obtained by photographing under the first photographing environment, based on the difference from the feature point;
Display control means for displaying the trajectory corrected by the trajectory correction means;
An image processing apparatus comprising:
<Claim 2>
The trajectory correction means includes
First setting means for setting two or more straight lines intersecting as first feature points in a moving image obtained by photographing under a first photographing environment by photographing means;
Trajectory acquisition means for acquiring a trajectory of a moving object from a moving image obtained by photographing in the first photographing environment by the photographing means;
Second setting means for setting two or more straight lines intersecting as second feature points in a moving image obtained by photographing under a second photographing environment different from the first photographing environment by the photographing means;
First shooting angle calculation means for calculating a shooting angle of the shooting means when shot under the first shooting environment from angles formed by two or more straight lines set by the first setting means;
Second shooting angle calculation means for calculating a shooting angle of the shooting means when shot under the second shooting environment from angles formed by two or more straight lines set by the second setting means;
A shooting angle difference calculating unit that calculates a difference between the shooting angle calculated by the first shooting angle calculating unit and the shooting angle calculated by the second shooting angle calculating unit;
Correction means for correcting the position of the trajectory acquired by the trajectory acquisition means based on the difference of the imaging angle calculated by the imaging angle difference calculation means;
With
The image processing apparatus according to claim 1, wherein the display control unit displays the locus corrected by the correction unit.
<Claim 3>
The trajectory correction means includes
Image designating means for designating the image to be moved;
A moving target image extracting means for extracting an image corresponding to an image designated by the image designating means from a moving image obtained by photographing in the first photographing environment by the photographing means;
With
The image processing apparatus according to claim 2, wherein the trajectory acquisition unit acquires the trajectory based on the image extracted by the movement target image extraction unit.
<Claim 4>
The trajectory correction means includes
Third setting means for setting a straight line as a third feature point in the moving image obtained by photographing in the first photographing environment by the photographing means;
Fourth setting means for setting a straight line as a fourth feature point in the moving image obtained by photographing in the second photographing environment by the photographing means;
A ratio calculating means for calculating a ratio of the length of the straight line set by the third setting means and the straight line set by the fourth setting means;
With
The image processing apparatus according to claim 2, wherein the correction unit further corrects the position of the trajectory acquired by the trajectory acquisition unit based on the ratio calculated by the ratio calculation unit.
<Claim 5>
The trajectory correction unit includes a position difference calculation unit that calculates a difference in position between the straight line set by the third setting unit and the straight line set by the fourth setting unit,
The image processing apparatus according to claim 4, wherein the correction unit further corrects the position of the trajectory acquired by the trajectory acquisition unit based on the position difference calculated by the position difference calculation unit. .
<Claim 6>
The first setting means sets two or more straight lines corresponding to moving images obtained by shooting at a plurality of shooting angles in a first shooting environment by the shooting means,
The locus acquisition means acquires the locus for each photographing,
The first shooting angle calculation means calculates a plurality of shooting angles,
The photographing angle difference calculating means selects the one closest to the photographing angle calculated by the second photographing angle calculating means from the plurality of photographing angles calculated by the first photographing angle calculating means, and is selected. Calculating a difference between the shooting angle and the shooting angle calculated by the second shooting angle calculation means;
The correction unit corresponds to a shooting angle selected by the shooting angle difference calculation unit from a plurality of trajectories acquired by the trajectory acquisition unit based on the difference of the shooting angle calculated by the shooting angle difference calculation unit. 6. The image processing apparatus according to claim 2, wherein the position of the object is corrected.
<Claim 7>
The image processing apparatus according to claim 2, wherein the trajectory acquisition unit sets a straight line as the movement target and acquires the trajectory of the straight line as the trajectory.
<Claim 8>
The trajectory correction unit includes a second trajectory acquisition unit that acquires a trajectory of a second movement target as a second trajectory from a moving image obtained by photographing in the second photographing environment by the photographing unit,
The display control means displays the second trajectory acquired by the second trajectory acquisition means together with the trajectory corrected by the trajectory correction means. The image processing apparatus described.
<Claim 9>
The trajectory correction unit includes a trajectory difference determination unit that determines a difference between the corrected trajectory and the second trajectory acquired by the second trajectory acquisition unit,
The image processing apparatus according to claim 8, wherein the display control unit causes display according to a determination result by the trajectory difference determination unit.
<Claim 10>
The display control means displays the corrected trajectory corresponding to a still image at a predetermined timing among moving images obtained by photographing in the first photographing environment by the photographing means. The image processing apparatus according to any one of claims 1 to 9.
<Claim 11>
An image processing method using an image processing apparatus,
A first feature point in a moving image obtained by photographing under a first photographing environment and a second feature in a moving image obtained by photographing under a second photographing environment different from the first photographing environment. A trajectory correction step for correcting a trajectory that is a moving object obtained from a moving image obtained by photographing in the first photographing environment, based on the difference from the feature point;
A display step for displaying the locus corrected in the locus correction step;
An image processing method comprising:
<Claim 12>
Computer
A first feature point in a moving image obtained by photographing under a first photographing environment and a second feature in a moving image obtained by photographing under a second photographing environment different from the first photographing environment. Trajectory correction means for correcting a trajectory that is a moving object obtained from a moving image obtained by photographing in the first photographing environment based on the difference from the feature point of
Display control means for displaying the locus corrected by the locus correcting means;
Program to function as.

100 Image processing system 1, 2, 3 Camera (photographing means)
DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 Control part (trajectory correction means)
11a 1st setting part 11b 2nd setting part 11c 1st locus acquisition part 11d 2nd locus acquisition part 11e 1st photography angle calculation part 11f 2nd photography angle calculation part 11g photography angle difference calculation part 11h correction part 11i 3rd setting part 11j Fourth setting unit 11k Ratio calculating unit 11m Position difference calculating unit 11n Trajectory difference determining unit 11p Image specifying unit 11r Moving target image extracting unit 11s Display control unit 13 Display unit (display means)

Claims (16)

Second moving image object that can be moved, which is taken under a different second imaging environment from the at least one first moving image and the first photographing environment of the shooting under the first imaging environment Image acquisition means for acquiring
A trajectory generating means for generating a first trajectory based on the motion of the subject included in the first moving image acquired by the image acquiring means;
Position information acquisition means for acquiring position information in the image based on feature points included in the movable subject of the first moving image and the second moving image;
A trajectory correcting unit that corrects the first trajectory generated by the trajectory generating unit based on the positional information acquired by the positional information acquiring unit so as to fit in the second imaging environment;
And synthesizing for synthesizing means such that the first trajectory is superimposed corrected by the trajectory compensation means to the second moving image,
An image processing apparatus comprising: Before SL trajectory correction means, the positional information based on the acquired device location information obtained by the correction to fit under a different second imaging environment from said first trajectory said first photographing environment To
The image processing apparatus according to claim 1. Setting means for setting a line feature points included in the subject that may be the movement of the front Symbol first moving image and the second moving image,
Further comprising
Prior Symbol position information acquiring unit, on the basis of the line set in the feature points included in the subject can be said moved by said setting means, it may be the movement of the first moving image and the second moving image object Get location information for
The image processing apparatus according to claim 2. Position information difference calculating means for calculating a difference between position information of the movable subject between the first moving image and the second moving image based on the position information acquired by the position information acquiring means. ,
The trajectory correction means, based on the difference of the position information calculated by the position information difference calculation detemir stage, fits under a different second imaging environment from the first said trajectory of the first photographing environment To correct,
The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus. The trajectory correction unit corrects the position of the first trajectory in a direction in which the difference of the position information is reduced based on the difference of the position information, thereby causing the first trajectory to be corrected in the second imaging environment. Correct to fit below,
The image processing apparatus according to claim 4. The trajectory generating means generates a second trajectory based on the motion of the subject in the second moving image,
A trajectory synthesizing unit that synthesizes the first trajectory corrected by the trajectory correcting unit and the second trajectory into the second moving image;
The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus. The trajectory generation means further comprises detection means for detecting the movement of the subject photographed in the first moving image in time series from a plurality of images constituting the first moving image,
The trajectory generating means generates a first trajectory based on an operation detected along the time series from the plurality of images by the detecting means.
The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus. Display control means for controlling the display means to display the first moving image and the second moving image;
Further comprising
The image processing according to any one of claims 2 to 7, wherein the display control unit causes the display unit to display the first trajectory or the second trajectory generated by the trajectory generation unit. apparatus. The display control means displays the first moving image and the second moving image in time series,
The trajectory generation means generates a trajectory image based on the operation detected by the detection means corresponding to the range displayed by the display control means,
The display control means causes the display means to display the trajectory image generated by the trajectory generation means.
The image processing apparatus according to claim 8. A difference calculating means for calculating a difference between the second locus generated by the locus generating means and the first locus corrected by the locus correcting means;
Informing means for informing information based on the difference calculated by the difference calculating means;
The image processing apparatus according to claim 6, further comprising: The notification means includes a threshold setting means for setting a threshold for determining whether or not to notify information based on the difference,
The image processing apparatus according to claim 10, wherein the notification unit notifies information based on the difference when the difference exceeds a threshold set by the threshold setting unit. A photographing angle calculating means for calculating a photographing angle when each of the first moving image and the second moving image is photographed;
12. The trajectory correction unit corrects the first trajectory so as to match a shooting angle of the second moving image based on a calculation result of the shooting angle calculation unit. The image processing apparatus according to any one of the above. The setting means sets a plurality of straight lines in the first moving image or the second moving image,
The shooting angle calculation means calculates the shooting angle based on an angle formed by a plurality of straight lines set by the setting means.
The image processing apparatus according to claim 12. A specifying means for specifying display start and end points in the first moving image and the second moving image;
Further comprising
The trajectory generating means generates a first trajectory in the first moving image based on the display start and end times specified by the specifying means,
The trajectory correction unit corrects the first trajectory generated by the trajectory generation unit so that the first trajectory is adapted to the second shooting environment in a range corresponding to the display start and end time points.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Second moving image object that can be moved, which is taken under a different second imaging environment from the at least one first moving image and the first photographing environment of the shooting under the first imaging environment An image acquisition step for acquiring,
A trajectory generating step for generating a first trajectory based on the motion of the subject included in the first moving image acquired by the image acquiring step;
A position information acquisition step of acquiring position information in the image based on feature points included in the movable subject of the first moving image and the second moving image;
Based on the position information acquired by the position information acquisition step , the first trajectory generated by the trajectory generation step is adapted to a second shooting environment different from the first shooting environment. A locus correction step to be corrected;
And synthesizing for synthesizing step such that the first trajectory is superimposed corrected by the trajectory compensation means to the second moving image,
An image processing method comprising: Computer
The subject can move the second moving image captured under a different second imaging environment from the at least one first moving image and the first photographing environment of that Tosa under the first imaging environment Image acquisition means to acquire,
Trajectory generating means for generating a first trajectory based on the motion of the subject included in the first moving image acquired by the image acquiring means;
Position information acquisition means for acquiring position information in the image based on feature points included in the movable subject of the first moving image and the second moving image;
Based on the position information acquired by the position information acquisition unit , the first trajectory generated by the trajectory generation unit is adapted to a second shooting environment different from the first shooting environment. Trajectory correction means for correcting,
Synthesis synthesizing means such that the first trajectory is corrected by the trajectory compensation means to the second moving image is superimposed,
A program characterized by functioning as

Images