KR101913667B1 - Golf swing analysis system using inertial sensor and Multiple cameras and Golf swing analysis method using the same - Google Patents

Abstract

The present invention relates to a golf swing analysis system using a multi-camera and a single inertial sensor, and a golf swing analysis method using the multi-camera and the single inertia sensor. The golf swing analysis system using the multi- An inertia sensor mounted on a user's hand for sensing a position of a user's hand during a golf swing, a depth camera for detecting a depth image obtained through the depth camera, A club calculating unit for detecting a motion of a golf club held by a user from a swing image photographed by the high frame camera; and a control unit for controlling the golf club based on the sensed information measured through the inertial sensor A hand calculator for calculating a motion of the user's hand at the time of swing The rain.
The golf swing analysis system using the multiple cameras and the single inertial sensor and the golf swing analysis method using the multiple cameras and the single inertial sensor according to the present invention is characterized in that an inertial sensor is provided to track a user's hand during a golf swing, The golfer can analyze the posture of the user during the swing, and can detect the movement of the golf club during the swing through the high frame camera, thereby obtaining a more accurate analysis result of the golf swing.

Description

[0001] The present invention relates to a golf swing analysis system using multiple cameras and a single inertial sensor, and a golf swing analysis method using the same.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a golf swing analysis system using multiple cameras and a single inertial sensor, and a golf swing analysis method using the system. The present invention relates to a golf swing analysis system using a multi-camera and a single inertial sensor for analyzing a golf swing, and a golf swing analysis method using the same.

Golf is a game in which a ball stopped on the course is hit by a golf club and placed in a designated hole, and competes with many more and fewer used balls. In order to be good at golf swing, it is necessary to understand various golf movements and to know and prevent wrong postures.

In golf swing practice, a good golf swing can only be achieved if each action is implemented correctly, from address to back swing, down swing, impact and finish. . In order to achieve such a good golf swing, it is necessary to repeatedly perform the calibration based on the analysis and the steady practice based on the accurate analysis of the professional golf swing.

Researches on various posture analysis simulation systems for improvement of current record and posture correction are being actively carried out.

As a conventional analysis system, in the case of a 2D analysis using a camera, a camera is placed on the front or side of a subject, and the state of swing progress is stored as an image, and then analyzed by an expert. In such a case, evaluation and analysis of the sports action with the human eye can be largely changed according to the subjectivity of the human being, and there is a disadvantage that it is difficult to analyze the joints that do not appear in the image.

In addition, in order to solve the disadvantages of such a two-dimensional analysis, researches on a motion analysis system using sensors of various types are actively conducted. However, in order to acquire the motion of the human body, it is inconvenient to attach at least 15 sensors to the body.

Open Patent Publication No. 10-2014-0044754: Golf swing analysis device and golf swing analysis method

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above problems, and an inertial sensor is provided to track a user's hand during a golf swing, and the user's swing posture is analyzed based on a depth image acquired through a depth camera And an object of the present invention is to provide a golf swing analysis system using a multi-camera and a single inertial sensor capable of detecting the movement of a golf club during a swing through a high frame camera, and a golf swing analysis method using the same.

According to an aspect of the present invention, there is provided a golf swing analysis system using a multi-camera and a single inertial sensor, including a depth camera for photographing a depth image of a user who performs a golf swing, An inertial sensor mounted on the user's hand for sensing the position of the user's hand during a golf swing, an attitude calculator for calculating attitude information of the user based on the depth image acquired through the depth camera, And a hand calculator for calculating the motion of the user's hand during the swing based on the sensed information measured through the inertial sensor

The attitude calculation unit may include a skeleton image generation unit that generates a skeleton image based on the depth image acquired from the depth camera and extracts coordinates corresponding to each joint of the user from the generated skeleton image, A coordinate extracting unit for extracting coordinates of a user's shoulders and feet from the coordinates and a state detecting unit for calculating a shoulder inclination of the user and a stance state of the feet based on the coordinates of the user's shoulders and feet extracted through the coordinate extracting unit do.

It is preferable that the state detecting section calculates the left and right widths of the user's feet and the front and back distance of the feet in a stance state of both feet.

Wherein the inertial sensor measures a three-axis acceleration value and a three-axis angular velocity value of a user's hand generated according to a golf swing, and the hand calculation unit calculates a three-axis acceleration value of the user's hand, .

The inertial sensor measures the three-axis acceleration value of the user's hand generated in accordance with the golf swing, and the golf swing analysis system using the multiple cameras and the single inertial sensor according to the present invention detects the swing based on the sensing information measured through the inertial sensor, A time calculating unit for calculating a top of swing time at which the head of the golf club is located at the top of the swing start point and the swing starting point, .

The point calculator includes an energy converter for converting a three-axis acceleration value of the inertial sensor generated according to a golf swing of a user into energy using a SVM (Signal Vector Magnitude) algorithm, and an energy conversion unit for converting the energy value converted through the energy converter A swing start point, a top of swing point, and a hit point.

Wherein the determination unit determines that the energy value converted through the energy conversion unit exceeds the reference value at a preset reference value as the swing start point, and when the energy value becomes equal to or less than the reference value after the swing start point, A time point at which the energy value reaches a maximum value after the top of swing time may be determined as the hit point.

Meanwhile, the golf swing analysis system using the multi-camera and the single inertia sensor according to the present invention calculates the time elapsed from the swing start point to the top of swing point to the top of swing point based on each point of view information calculated through the point- And a swing tempo calculation unit for calculating a ratio of an elapsed time from the swing time to the shot time.

Wherein the club calculating unit calculates a difference between a reference image which is a video of a current frame in a swing image obtained through the high frame camera and a first previous image which is an image of a previous frame of the reference image, A second image generating unit for generating a difference image using the difference between the first previous image and the second previous image, which is a previous frame of the first previous image, in the swing image; An analysis image generating unit for detecting an overlapping portion of the difference images generated through the first and second difference image generators to generate an analysis image; Or an image detecting section for detecting the head.

The club calculating unit may further include a noise removing unit for applying a morphology operation to the analysis image so as to remove noise of the analysis image generated through the analysis image generating unit.

Preferably, the image detecting unit detects the golf club in the analysis image using a Hough Transform, and performs curve fitting to prevent erroneous detection of the head in the detected golf club.

Meanwhile, a golf swing analysis method using a multi-camera and a single inertial sensor according to the present invention includes a depth image acquiring step of acquiring a depth image for a user performing a golf swing from a depth camera, A swing image acquiring step of acquiring a swing image of the user's swing during the golf swing from the high frame camera; A hand information calculating step of calculating a motion of a user's hand in a golf swing based on the sensed information of the inertia sensor acquired through the sensor information acquiring step, Based on the swing image acquired through the swing image acquisition step, And a club information calculation step of calculating the profile of the shaft or club head movements.

Preferably, the inertial sensor measures a three-axis acceleration value of a user's hand caused by a golf swing.

The golf swing analysis method using the multiple cameras and the single inertial sensor according to the present invention is characterized in that the golf swing start point, the swing start point, and the swing start point are determined based on the sensed information measured through the inertial sensor, And a time calculating step of calculating a top of swing time point at which the golf club head is located at the top side between the hit points.

The point calculation step may include an energy conversion step of converting the three-axis acceleration value of the inertial sensor generated according to the golf swing of the user into energy using a SVM (Signal Vector Magnitude) algorithm, The swing start point, the top of swing point, and the batting point in accordance with the energy value of the golf swing of the golf swing.

Wherein the determining step determines the time when the energy value converted through the energy conversion step exceeds the reference value at a preset reference value as the swing start time and the energy value is less than the reference value after the swing start time It is preferable to determine the time point as the top of swing time point and to determine the time point at which the energy value becomes the maximum value after the top of swing time point as the hit point.

Meanwhile, the golf swing analysis method using the multi-camera and the single inertia sensor according to the present invention may further include a step of calculating the time elapsed from the swing start point to the top of swing point, And a swing tempo calculating step of calculating a ratio of an elapsed time from the time of swing to the time of hitting.

Wherein the attitude information calculation step includes a skeleton image generation step of generating a skeleton image based on the depth image acquired from the depth camera and extracting coordinates corresponding to each joint of the user from the generated skeleton image, Extracting coordinates of a user's shoulders and feet from the coordinates extracted through the coordinate extracting step, and calculating a stance state of the user's shoulder inclination and both feet based on coordinates of a user's shoulder and feet extracted through the coordinate extracting step And a state detecting step.

The club information calculation step may include a step of generating a difference image using the difference between the reference image, which is the image of the current frame, and the first previous image, which is the image of the previous frame of the reference image, A second image generation step of generating a difference image using the difference between the first previous image and the second previous image, which is a previous frame of the first previous image, in the swing image; An analysis image generation step of detecting an overlapping part of the difference images generated through the first and second difference image generation steps to generate an analysis image; And a head of the golf club.

The club information calculating step may further include a noise removing step of applying a morphology operation to the analysis image to remove noise from the analysis image generated through the analysis image generating step.

In the image detection step, it is possible to detect the golf club in the analysis image using a Hough Transform, and perform curve fitting to prevent erroneous detection of the head part of the golf club in the detected golf club desirable.

The golf swing analysis system using the multiple cameras and the single inertial sensor and the golf swing analysis method using the multiple cameras and the single inertial sensor according to the present invention is characterized in that an inertial sensor is provided to track a user's hand during a golf swing, The golfer can analyze the posture of the user during the swing, and can detect the movement of the golf club during the swing through the high frame camera, thereby obtaining a more accurate analysis result of the golf swing.

1 is a conceptual diagram of a golf swing analysis system using a multi-camera and a single inertial sensor according to the present invention,
FIG. 2 is a block diagram of a golf swing analysis system using the multi-camera and single inertial sensor of FIG. 1,
FIG. 3 is a block diagram of an attitude calculation unit of the golf swing analysis system using the multiple cameras and the single inertial sensor of FIG. 1,
FIG. 4 is a block diagram of a point calculation unit of the golf swing analysis system using the multi-camera and single inertial sensor of FIG. 1,
FIG. 5 is a block diagram of a club calculation unit of the golf swing analysis system using the multi-camera and single inertial sensor of FIG. 1,
FIG. 6 is an illustration of a skeleton image displayed with coordinates generated by the skeleton image generation unit of the golf swing analysis system using the multi-camera and single inertial sensor of FIG. 1,
FIG. 7 is a photograph and a graph of the results detected by the state detector of the golf swing analysis system using the multi-camera and single inertial sensor of FIG. 1,
FIG. 8 is a graph showing the trajectory of a hand calculated by the hand calculator of the golf swing analysis system using the multi-camera and single inertial sensor of FIG. 1,
FIG. 9 is a graph of energy values converted by the energy conversion unit of the golf swing analysis system using the multiple cameras and the single inertial sensor of FIG. 1,
FIG. 10 is an image generated by the first-order image generating unit, the second-order image generating unit, and the analyzed image generating unit of the golf swing analysis system using the multi-camera and single inertial sensor of FIG. 1,
FIG. 11 is an analytical image in which noises are removed by the noise elimination of the golf swing analysis system using the multiple cameras and the single inertial sensor of FIG. 1,
FIG. 12 is an image of a golf swing analysis system using a multi-camera and a single inertial sensor of FIG. 1,
FIG. 13 is an image of a golf swing analysis system using a multi-camera and a single inertial sensor of FIG. 1,
FIG. 14 is a graph showing the trajectory of a golf club detected through the image detecting unit of the golf swing analysis system using the multiple cameras and the single inertial sensor of FIG. 1;

Hereinafter, a golf swing analysis system using multiple cameras and a single inertial sensor according to an embodiment of the present invention and a golf swing analysis method using the system will be described in detail with reference to the accompanying drawings. The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It is to be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing. In the accompanying drawings, the dimensions of the structures are enlarged to illustrate the present invention in order to clarify the present invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a part or a combination thereof is described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

1 to 5 show a golf swing analysis system using multiple cameras and a single inertial sensor according to the present invention.

Referring to the drawings, a golf swing analysis system 100 using a multi-camera and a single inertial sensor includes a depth camera 110 for photographing a depth image of a user swinging a golf swing, a depth camera 110 for photographing a golf swing, A camera 130, an inertial sensor 120 mounted on the user's hand to sense the position of the user's hand during a golf swing, a depth camera 110, a high frame camera 130 and an inertial sensor 120 And an information processing unit 200 for analyzing the user's golf swing based on the acquired information.

The depth camera 110 is installed at a predetermined distance from the front of the user so as to capture a depth image of the user. Preferably, the depth camera 110 is a device such as a Kinect device including a camera and a depth sensor capable of acquiring a color image and a depth image.

The high frame camera 130 is an RGB (Red Green Blue) camera capable of capturing an image with a high frame, and photographs the front of the user at a position adjacent to the depth camera 110. The high frame camera 130 captures the motion of the golf club during a golf swing.

At this time, the golf club has a shaft provided with a handle that can be held by a user at one end, and a head formed at the other end of the shaft to hit the golf ball. Since the golf club is a conventional golf club generally used in golf exercise, a detailed description will be omitted.

The inertial sensor 120 is mounted on a glove worn by a user or attached to a user's hand, and is a sensor for measuring a three-axis acceleration value of a user's hand caused by a golf swing. The inertial sensor 120 includes a gyro sensor capable of measuring three-axis angular velocity and an acceleration sensor capable of measuring three-axis acceleration in order to track movement of the hand.

The gyro sensor can obtain angular velocity data of x, y, and z axes, and is expressed by Equation 1 below. The acceleration sensor can obtain acceleration data of x, y, and z axes, and is expressed by Equation 2 below.

The inertial sensor 120 is a sensor for measuring a three-axis acceleration value which is conventionally used, and thus a detailed description thereof will be omitted.

The information processing unit 200 includes an attitude calculation unit 210 for calculating attitude information of a user on the basis of the depth image acquired through the depth camera 110, A hand calculator 230 for calculating a motion of the user's hand in a swing based on the sensed information measured through the inertial sensor 120, A swing tempo calculating unit 240 for calculating swing tempos based on the sensed information measured through the inertial sensor 120 and a swing tempo calculating unit 240 for calculating a swing tempo based on the time information calculated through the time calculating unit 240 And a swing tempo calculation unit 250.

The attitude calculating unit 210 measures the shoulder inclination of the user and the stance state of the feet during golf swing and includes a skeleton image generating unit 211, a coordinate extracting unit 212, and a state detecting unit 213.

The skeleton image generation unit 211 generates a skeleton image based on the depth image acquired from the depth camera 110 and extracts coordinates corresponding to each joint of the user from the generated skeleton image. The skeleton image generation unit 211 extracts the user's body from the depth image provided from the depth camera 110. At this time, the skeleton image generation unit 211 generates joint coordinates in the body range of the extracted depth image as shown in FIG. 6 by applying joint tracking software capable of recognizing 20 major joints of a person.

The coordinate extraction unit 212 extracts the coordinates of the user's shoulders and feet from the coordinates extracted through the skeleton image generation unit 211. [

The state detecting unit 213 calculates the shoulder inclination of the user and the stance state of the feet on the basis of the coordinates of the shoulders and feet of the user extracted through the coordinate extracting unit 212. 7 is a photograph and a graph of the shoulder inclination calculated by the state detecting unit 213. As shown in Fig. As shown in the figure, the state detector 213 extracts the coordinates of the left and right shoulders, calculates the slope of the line passing through the shoulder coordinates, and calculates the slope of the user's shoulder.

The state detecting unit 213 calculates the stance state of the user's feet. The state detecting unit 213 calculates the left and right widths of the user's feet or the front and rear distance of the feet based on the coordinates of the left and right feet of the user extracted through the coordinate extracting unit 212.

The hand calculator 230 calculates the movement trajectory of the user's hand on the basis of the three-axis acceleration value of the user's hand measured from the inertial sensor 120. The hand calculator 230 tracks the position of the user's hand based on the three-axis angular velocity data and the three-axis acceleration data of the user's hand measured through the inertial sensor 120. [ A method of calculating the trajectory of the hand through the data of the inertial sensor 120 in the hand calculator 230 will be described in detail as follows.

시간에 따라 변화하는 쿼터니언 정보를 얻기 위해서는 가속도 데이터를 이용하여 중력의 방향 정보를 얻어야 하지만, 초기 입력되는 가속도 데이터는 불필요한 데이터와 비교적 많은 데이터가 포함되어 있기 때문에 정규화 작업이 필요하며, 정규화 작업이 진행된 가속도 데이터는

로 표현된다. 정규화된 가속도와 초기 쿼터니언 데이터를 하기의 수학식 3에 대입하여 중력의 예상 움직임 방향을 추출한다. The data of the angular velocity and the acceleration obtained through the inertial sensor 120 are used to make a quaternion. Since there is no initial quaternion data, an arbitrary initial setting value is input.

In order to obtain quaternion information that varies with time, the direction information of gravity must be obtained using the acceleration data. However, since the acceleration data initially input includes unnecessary data and a relatively large amount of data, a normalization operation is required. The acceleration data

Lt; / RTI > The normalized acceleration and the initial quaternion data are substituted into Equation (3) to extract the expected movement direction of gravity.

각속도와 초기 쿼터니언을 이용하여 시간에 따라 증가하는 값을 추정함으로써 변해가느 속도를 알 수 있는 쿼터니언의 발산지수인

를 출력한다. 다음, 발산지수인 β와 커터니언q 그리고 샘플링 주기 Δt를 곱하여 하기의 수학식 4와 같이 쿼터니언 정보를 추출한다. The output of the gravity direction data is multiplied by the sampling period and the angular velocity to obtain an angular velocity change amount with respect to time.

By using the angular velocity and the initial quaternion, the quaternion's divergence index

. Next, the quaternion information is extracted by multiplying the divergence index [beta], the cutterance q, and the sampling period [Delta] t, as shown in Equation 4 below.

Since the extracted quaternion data can not be directly computed, it is converted into a rotation matrix and computed using the following equation (5).

)을 계산하고, 이를 이용하여 속도 (

)를 구한다. 출력된 속도 데이터는 드리프트 현상을 줄이기 위해 하이패스필터를 통해 오류를 줄이고 초기 설정된 위치 좌표에

와 같이 계산하여 위치를 추적한다. In order to track the position, quaternion rotation information and velocity information are required. In order to calculate the velocity, the rotation matrix and the inverse matrix of the acceleration data are multiplied to calculate the tilt error compensation

) Is calculated, and the velocity (

). The output velocity data reduces the error through the high pass filter to reduce the drift phenomenon,

And the position is tracked.

The hand calculator 230 tracks the trajectory of the user's hand during the golf swing and provides the graph as shown in FIG.

Based on the sensed information measured through the inertial sensor 120, the time calculating unit 240 calculates a time point at which the golf club starts to hit the golf ball, And a determination unit 242. The energy conversion unit 241 and the determination unit 242 calculate the top of swing time and finish time of the swing.

The energy conversion unit 241 converts the three-axis acceleration value of the inertial sensor 120 generated according to the user's golf swing into energy using a SVM (Signal Vector Magnitude) algorithm. The energy conversion unit 241 converts the three-axis acceleration data provided from the inertial sensor 120 into energy E using the following equation (6), and the value is displayed as shown in FIG.

The determination unit 242 determines the swing start point, the top of swing point, and the hit point based on the energy value converted through the energy conversion unit 241. Generally, in the golf swing, the energy value is kept close to 1 at the address posture in the ready state before the swing. The determination unit 242 determines a time when the energy value converted through the energy conversion unit 241 exceeds the reference value at a preset reference value as the backswing time, that is, the swing start time. At this time, the reference value may be 1 or a value of 0.5 to 1.5. The reference value can be set after calculating the energy value of the address posture according to the user. It is preferable to determine the address state before the swing start point.

In addition, since the swing starts when the top os swing reaches the swing starting point, there is almost no movement, so the energy value has a value close to zero. Therefore, the determination unit 242 determines the time point at which the energy value becomes equal to or less than the reference value after the swing start time point as the top of swing time point.

Then, the energy value becomes maximum at the moment of hitting the golf ball after the top of swing time. Therefore, the determining unit 242 determines a point in time at which the energy value reaches a maximum value after the top of swing time as the hit point. When the swing is completed after the hit point, the energy value approaches the reference value again, so the determining unit 242 determines a point in time at which the energy value becomes equal to or less than the reference value after the hit point as the finish time point. In addition, the determination unit 242 displays the identified points on the graph of the hand trajectory and the energy change graph, and provides the graph to the user.

The swing tempo calculator 250 calculates the swing tempo based on the time information from the swing starting point to the top of swing point and the time from the top of swing point to the impact point Of the elapsed time is calculated. That is, the swing tempo calculator 250 calculates the swing tempo according to the following equation (7).

Here, BS _time is the elapsed time from the swing start point to the top of swing point, and DS _time is the elapsed _time from the top of swing point to the hit point. At this time, the ideal swing tempo is 3: 1, and the swing tempo calculator 250 provides the user with the ideal swing tempo value and the calculated swing tempo information.

The club calculating unit 220 detects the movement of the golf club held by the user from the swing image photographed from the high frame camera 130 and includes a first image generating unit 221, 222, an analysis image generating unit 223, a noise removing unit 224, and an image detecting unit 225.

The first image generating unit 221 converts the swing image provided from the high frame camera 130 into a gray image and then outputs the reference image, which is the image of the current frame in the swing image, 1 The difference image is generated using the difference of the previous image. At this time, the first-order image generating unit 221 extracts mutually changed portions of the reference image and the first previous image to generate a difference image, and uses the following Equation (8).

Here, Gray _t is the reference image, Gray _{t - 1} is the first previous image, and Diff ₁ is the difference image generated by the first image generating unit 221.

The second image generating unit 222 generates a difference image using the difference between the first previous image and the second previous image, which is the image of the previous frame of the first previous image, in the swing image converted into the gray image . At this time, the second image generating unit 222 extracts mutually changed parts from the first previous image and the second previous image, and generates a difference image using the following equation (9).

Here, Gray _{t - 1} is the first previous image, Gray _{t -2} is the second previous image, and Diff ₂ is the difference image generated by the second image generating unit 222.

)을 이용한다. The analysis image generation unit 223 detects the overlapping portions of the difference images generated through the first and second difference image generation units 222 to generate an analysis image. The analysis image generation unit 223 generates an analysis image by using an And operation (

).

10, images generated by the primary image generation unit 221, the secondary image generation unit 222, and the analysis image generation unit 223 are posted. Here, the t image is the reference image, the t-1 image is the first previous image, the t-2 image is the second previous image, the difference image 1 and the difference image 2 are generated by the first- And the And image is an analysis image generated by the analysis image generation unit 223. The second image generation unit 222 generates the second image.

The noise removing unit 224 filters the noise of the analysis image generated through the analysis image generating unit 223. The analysis image generated by the analysis image generation unit 223 detects only a portion in the original image, but noise is generated in the image. The noise removal unit 224 performs a morphology operation on the analysis image. To remove noise.

Morphology operation is a method for extracting image features useful for expressing or describing region shapes such as boundaries and skeletons in image processing. It is useful for processing overlapping and noise of edges or regions, Closing, and opening.

으로 표현된다. 여기서 A는 분석영상이고, B는 모폴로지 연산의 시행에 필요한 구조요소 3X3배열입니다. 침식연산은 팽창과 반대로 밝은 영역을 줄이고, 어두운 영역을 늘려가는 연산으로

으로 표현된다. 열기와 닫힘연산은 침식과 팽창 연산을 이용하여 구현할 수 있는데, 열기 연산은 일반적으로 객체의 돌출한 부위를 제거해주는 역할과 잡음을 제거하는 역할로

으로 표현된다. 닫힘연산은 외곽선 부분들을 메우고, 비워진 픽셀들을 제거해서 부드럽게 만들어주며,

으로 표현된다. The dilation operation is an operation that uses a pixel around an image to increase a bright area,

. Where A is the analytical image and B is the 3X3 array of structural elements required for the execution of the morphology operation. The erosion operation is an operation that reduces the light area and increases the dark area as opposed to the expansion.

. Open and close operations can be implemented using erosion and expansion operations. Open operations generally remove the protruding parts of objects and remove noise.

. The close operation fills the outline portions, removes the emptied pixels to make them smooth,

.

The noise removing unit 224 extracts a smooth outline through the opening operation during the morphology operation and removes the noise. 11, an analytical image in which noise is removed by the noise removing unit 224 is added.

The image detecting unit 225 detects a shaft or a head of the golf club in the analysis image generated by the analysis image generating unit 223. At this time, the image detector 225 detects the shaft or the head of the golf club from the noise-removed analytical image by the noise eliminator 224.

The image detecting unit 225 performs a Kenny operation, which is one of edge detection, on the analyzed image. The Kenny operator is an operation required to perform Hough transform by a method of classifying an edge when a change in brightness value is larger than a slope of neighboring pixels in comparison with a neighboring pixel in a direction in which a change in brightness value is largest. 12 shows an image obtained by detecting a kenny edge in the analysis image.

The image detecting unit 225 uses the Hough transform to find an expressible straight line using the extracted edge image. The Hough transform is an algorithm for finding a straight line by mapping the y-x slope in the image to the intercept space b-a. In Fig. 13, the straight line detection result is obtained using the Hough transform in the analytical image from which noise has been removed. The straight line detected in Fig. 13 represents the shaft of the golf club, and the end of the straight line can be used to estimate the golf swing trajectory.

On the other hand, the image detecting unit 225 detects the shaft or head of the golf club using curve fitting to prevent erroneous detection of straight line detection by Hough transform. Curve fitting refers to the task of matching image images to an appropriate curve, through which a function of a mathematical model is searched to deduce missing data and to fill in the appropriate data in the empty space between the data and the data. Regression analysis and interpolation are widely used for curve fitting. Regression analysis is used when the data contains significant size errors or scatter data, and there are high order polynomials, Fourier, Gaussian, and sine. The interpolation method is used when the data approximates to the actual data, such as B-spline, cubic spline, linear interpolation, and recent interpolation.

14 is a graph showing the trajectory of the golf club detected through the image detecting unit 225. In FIG. Referring to the drawings, it can be seen that a smoother swing trajectory is estimated by the curve fitting.

The information processing unit 200 includes an attitude calculating unit 210, a club calculating unit 220, a hand calculating unit 230, a time calculating unit 240, and a swing tempo calculating unit 250, And a display unit for displaying a result calculated by the display unit. It is preferable that a display device such as an LCD is applied to the display part.

The golf swing analysis method using the golf swing analysis system 100 using the multiple cameras and the single inertial sensor constructed as described above will be described in detail as follows.

The golf swing analysis method according to the present invention includes a depth image acquisition step, a sensor information acquisition step, a swing image acquisition step, a posture information calculation step, a hand information calculation step, a time calculation step, a swing tempo calculation step, and a club information calculation step .

The depth image acquiring step is a step of acquiring a depth image for a user performing a golf swing from the depth camera 110. The depth camera 110 mounted on the front of the user captures a depth image of the user and transmits the depth image to the information processing unit 200 when the user performs a golf swing.

The sensor information acquiring step is a step of acquiring sensing information of the inertial sensor 120 mounted on the user's hand during the golf swing. The inertial sensor 120 transmits the three-axis angular velocity data and the three-axis acceleration data of the hand when the glove or the user's hand is worn in the user's hand and swings, to the information processing unit 200.

The swing image acquiring step is a step of acquiring a swing image for the swing of the user in the golf swing from the high frame camera 130. The high frame camera 130 installed at a position adjacent to the depth camera 110 photographs a user at the time of swing, and the photographed swing image is transmitted to the information processing unit 200.

It is preferable that the depth image acquiring step, the sensor information acquiring step, and the swing image acquiring step described above are performed at the same time during the golf swing.

The posture information calculation step is a step of calculating posture information of a user based on the depth image acquired through the depth image acquisition step, and includes a skeleton image generation step, a coordinate extraction step, and a state detection step.

The skeleton image generation step is a step of generating a skeleton image based on the depth image acquired from the depth camera 110 and extracting coordinates corresponding to each joint of the user from the generated skeleton image. The skeleton associative generation unit generates a skeleton image or a body skeleton image on the depth image provided from the depth camera 110, and generates joint coordinates of a human in the skeleton image by the joint tracking software.

The coordinate extracting step extracts coordinates of the user's shoulders and feet of the coordinates extracted through the skeleton image generating step. The coordinate extraction unit 212 extracts the coordinates of the left and right shoulders and the coordinates of the left and right feet of the user in the joint coordinates of the skeleton image.

The state detecting step is a step of calculating the shoulder inclination of the user and the stance state of the feet on the basis of the coordinates of the shoulders and the feet of the user extracted through the coordinate extracting step. The state detecting unit 213 calculates the inclination of the shoulders, the left and right widths of the feet, and the front and back distance of the feet based on the coordinates extracted from the coordinate extracting unit 212 as described above. The resultant value calculated by the state detecting unit 213 is displayed to the user through the display unit.

The hand information calculating step is a step of calculating the motion of the user's hand during the golf swing based on the sensing information of the inertial sensor 120 acquired through the sensor information acquiring step. The hand calculator 230 calculates the movement trajectory of the user's hand based on the three-axis acceleration value and the three-axis angular velocity value of the user's hand measured from the inertial sensor 120. The hand calculation unit 230 displays the calculated movement trajectory of the hand to the user through the display unit.

The point calculating step may include a swing start point, a golf club head hit point, and a swing start point and a hit point based on the sensed information measured through the inertial sensor 120, Calculating a top of swing time point located on the uppermost side, and includes an energy conversion step and a discrimination step.

The energy conversion step converts the three-axis acceleration value of the inertial sensor 120 generated according to the user's golf swing into energy using a SVM (Signal Vector Magnitude) algorithm. The energy conversion unit 241 of the point-of-view calculation unit 240 converts the three-axis acceleration value of the inertial sensor 120 generated according to the golf swing into energy, and calculates a graph of a change in the energy value with time And displayed on the display unit.

The discriminating step discriminates the swing start point, the top of swing point and the hit point according to the energy value according to the golf swing of the user converted through the energy conversion step. The determination unit 242 of the time point calculation unit 240 determines the swing start point, the top of swing point, and the hit point based on the energy value converted through the energy conversion unit 241 as described above. At this time, the determination unit 242 determines that the energy value converted through the energy conversion step exceeds the reference value at the preset reference value as the swing start point, and after the swing start point, Of the swing time point is determined to be the top of swing time point, and the time point at which the energy value reaches the maximum value after the top of swing time point is determined as the hitting time point.

The determination unit 242 displays the calculated points on the graph calculated by the energy conversion unit 241 and the graph calculated by the hand calculation unit 230, and displays the calculated points on the display unit.

The swing tempo calculating step may calculate the swing tempo based on the time information calculated from the swing starting point to the top of swing time and the elapsed time from the top of swing time to the shooting time, . The swing tempo calculator 250 calculates the swing tempo based on the time information from the swing starting point to the top of swing point and the time from the top of swing point to the impact point Of the elapsed time is calculated.

The club information calculating step may include calculating a motion of a shaft or a head of a golf club held by a user during a golf swing based on the swing image acquired through the swing image acquiring step, A generating step, an analysis image generating step, a noise removing step, and an image detecting step.

The first image generating step generates the first image using the difference between the reference image, which is the image of the current frame, and the first previous image, which is the image of the previous frame of the reference image, from the swing image acquired through the high frame camera 130, . The first image generating unit 221 converts the swing image provided from the high frame camera 130 into a gray image and then outputs the reference image, which is the image of the current frame in the swing image, 1 The difference image is generated using the difference of the previous image.

The second image generation step is a step of generating a difference image using the difference between the first previous image and the second previous image, which is a previous frame of the first previous image, in the swing image. The second image generating unit 222 generates a difference image using the difference between the first previous image and the second previous image, which is the image of the previous frame of the first previous image, in the swing image converted into the gray image .

The analysis image generation step is a step of generating an analysis image by detecting overlapping portions of the difference images generated through the first and second difference image generation steps. The analysis image generation unit 223 generates an analysis image by detecting overlapping portions of the difference images generated through the first and second difference image generation units 222 using the And association.

The noise removing step is a step of removing noise from the analysis image generated through the analysis image generating step. At this time, the noise removing unit 224 removes noise by applying a morphology operation to the analyzed image.

The image detection step is a step of detecting the head of the golf club or the shaft of the golf club in the analysis image generated in the analysis image generation step. At this time, the golf club is detected in the analysis image using Hough Transform through the image detection unit 225, and the curve fitting is performed to prevent the erroneous detection of the head portion of the golf club in the detected golf club . The image detecting unit 225 displays the detected result to the user through the display unit.

As described above, the golf swing analysis system 100 and the golf swing analysis method using the multiple cameras and the single inertial sensor according to the present invention are provided with the inertial sensor 120 for tracking the user's hand in the golf swing The golfer can analyze the user's swing attitude based on the depth image acquired through the depth camera 110 and detect the movement of the golf club during the swing through the high frame camera 130. Therefore, Can be obtained.

The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features presented herein.

100: Golf swing analysis system using multiple cameras and single inertial sensor
110: Depth camera
130: high frame camera
120: inertia sensor
200: Information processing unit
210:
211: Skeleton image generation unit
212: Coordinate extraction unit
213:
220: Club calculation department
221: a first image generating unit
222: Secondary image generating unit
223: Analysis image generating section
224: Noise canceling
225:
230: Hand calculation unit
240:
241: energy conversion section
242:
250: Swing tempo calculation unit

Claims (20)

A depth camera for photographing a depth image of a user swinging the golf swing;
A high frame camera for photographing the golf swing of the user;
An inertial sensor mounted on a user's hand for sensing a position of a user's hand during a golf swing, the inertial sensor measuring a three-axis acceleration value of a user's hand generated according to a golf swing;
An attitude calculation unit for calculating attitude information of a user based on the depth image acquired through the depth camera;
A club calculating unit for detecting a motion of a golf club held by a user from the swing image photographed by the high frame camera;
A hand calculator for calculating a movement of the user's hand in a swing based on the sensed information measured through the inertial sensor;
A swing start point, a swing start point, a swing start point, and a swing start point, wherein the swing start point, the golf club hit the golf ball, and the swing start point and the hit point, based on the sensed information measured through the inertial sensor, and a time calculating unit for calculating a top of swing time,
The point-
An energy conversion unit for converting the three-axis acceleration value of the inertial sensor generated according to the golf swing of the user into energy using a SVM (Signal Vector Magnitude) algorithm;
And a determination unit for determining the swing start point, the top of swing point, and the hit point based on the energy value converted through the energy conversion unit.
Golf swing analysis system using multiple cameras and single inertial sensor.
The method according to claim 1,
The attitude calculation unit
A skeleton image generating unit for generating a skeleton image based on the depth image acquired from the depth camera and extracting coordinates corresponding to each joint of the user from the generated skeleton image;
A coordinate extraction unit for extracting coordinates of the user's shoulders and feet from the coordinates extracted through the skeleton image generation unit;
And a state detector for calculating the shoulder inclination of the user and the stance state of the feet based on coordinates of the user's shoulders and feet extracted through the coordinate extracting unit,
Golf swing analysis system using multiple cameras and single inertial sensor.
3. The method of claim 2,
Wherein the state detecting section is a stance state of a pair of legs, the state detecting section calculating a lateral distance of the user's legs and a front-
Golf swing analysis system using multiple cameras and single inertial sensor.
The method according to claim 1,
Wherein the inertia sensor measures a three-axis acceleration value and a three-axis angular velocity value of a user's hand generated according to a golf swing,
Wherein the hand calculation unit calculates a movement trajectory of the user's hand based on the three-axis acceleration value of the user's hand measured from the inertial sensor,
Golf swing analysis system using multiple cameras and single inertial sensor.
delete delete The method according to claim 1,
The determination unit
Wherein the energy conversion unit determines the time when the energy value converted by the energy conversion unit exceeds the reference value at a preset reference value as the swing start time, and when the energy value becomes less than the reference value after the swing start time, And determining a point in time when the energy value becomes a maximum value after the top of swing time as the point of time of the hitting,
Golf swing analysis system using multiple cameras and single inertial sensor.
The method according to claim 1,
Calculating a ratio of an elapsed time from the swing start point to the top of swing point to an elapsed time from the top of swing point to the hit point based on each viewpoint information calculated through the viewpoint calculating unit; ; ≪ / RTI >
Golf swing analysis system using multiple cameras and single inertial sensor.
The method according to claim 1,
The club calculator
A first image generating unit for generating a difference image using a difference between a reference image, which is a current frame image, and a first previous image, which is a previous frame image of the reference image, in a swing image acquired through the high frame camera; ;
A second image generation unit for generating a difference image using the difference between the first previous image and the second previous image, which is an image of a previous frame of the first previous image, in the swing image;
An analysis image generating unit for detecting an overlapping portion of the difference images generated through the first and second difference image generating units to generate an analysis image;
And an image detecting unit for detecting a shaft or a head of the golf club from the analysis image generated by the analysis image generating unit.
Golf swing analysis system using multiple cameras and single inertial sensor.
10. The method of claim 9,
Wherein the club calculator further comprises a noise eliminator for applying a morphology operation to the analysis image to filter the noise of the analysis image generated through the analysis image generator,
Golf swing analysis system using multiple cameras and single inertial sensor.
10. The method of claim 9,
Wherein the image detecting unit detects the golf club in the analysis image using a Hough Transform and performs curve fitting to prevent erroneous detection of the head in the detected golf club.
Golf swing analysis system using multiple cameras and single inertial sensor.
A depth image acquiring step of acquiring a depth image for a user performing a golf swing from a depth camera;
A sensor information acquiring step of acquiring sensed information of an inertia sensor mounted on a user's hand during the golf swing and measuring a three-axis acceleration and a three-axis angular velocity of a user's hand generated according to a golf swing;
A swing image acquiring step of acquiring a swing image of the user's swing in the golf swing from the high frame camera;
An attitude information calculating step of calculating attitude information of a user based on the depth images acquired through the depth image acquiring step;
A hand information calculating step of calculating a movement of a user's hand in golf swing based on the sensed information of the inertial sensor acquired through the sensor information acquiring step;
A club information calculating step of calculating a movement of a shaft or a head of a golf club held by the user during a golf swing based on the swing image acquired through the swing image acquiring step;
A swing start point, a golf swing start point, a swing start point, and a swing start point, wherein the swing start point, the golf club head hit a golf ball, and the top of swing and calculating a top of swing time,
The point calculation step
An energy conversion step of converting the three-axis acceleration value of the inertial sensor generated according to the golf swing of the user into energy using a SVM (Signal Vector Magnitude) algorithm;
A swing start point, a top of swing point, and a hit point according to an energy value according to a golf swing of the user converted through the energy conversion step;
A golf swing analysis method using multiple cameras and a single inertial sensor.
delete delete 13. The method of claim 12,
Wherein the determining step determines the time when the energy value converted through the energy conversion step exceeds the reference value at a preset reference value as the swing start time and the energy value is less than the reference value after the swing start time Determining a point of time at which the energy value becomes a maximum value after the top of swing time as the point of time of the swing,
A golf swing analysis method using multiple cameras and a single inertial sensor.
13. The method of claim 12,
Calculating a swing tempo computing a ratio of an elapsed time from the swing start point to the top of swing point to an elapsed time from the top of swing point to the hit point based on each viewpoint information calculated through the point calculating step; Further comprising:
A golf swing analysis method using multiple cameras and a single inertial sensor.
13. The method of claim 12,
The posture information calculating step
A skeleton image generating step of generating a skeleton image based on the depth image acquired from the depth camera and extracting coordinates corresponding to each joint of the user from the generated skeleton image,
A coordinate extracting step of extracting a coordinate of a user's shoulder and foot from coordinates extracted through the skeleton image generating step,
And a state detecting step of calculating the shoulder inclination of the user and the stance state of the feet based on the coordinates of the shoulders and the feet of the user extracted through the coordinate extracting step,
A golf swing analysis method using multiple cameras and a single inertial sensor.
13. The method of claim 12,
The club information calculating step
A first image generation step of generating a difference image using a difference between a reference image which is a current frame image and a first previous image which is a previous frame image of the reference image in a swing image obtained through the high frame camera, ,
A second image generation step of generating a difference image using the difference between the first previous image and the second previous image, which is a previous frame of the first previous image, in the swing image;
An analysis image generation step of generating an analysis image by detecting overlapping parts of the difference images generated through the first and second difference image generation steps,
And an image detecting step of detecting the head of the golf club or the shaft of the golf club in the analysis image generated in the analysis image generating step,
A golf swing analysis method using multiple cameras and a single inertial sensor.
19. The method of claim 18,
The club information calculating step
Further comprising a noise removal step of applying a morphology operation to the analysis image to remove noise from the analysis image generated through the analysis image generation step,
A golf swing analysis method using multiple cameras and a single inertial sensor.
19. The method of claim 18,
Detecting the golf club in the analysis image using a Hough Transform in the image detection step and performing curve fitting to prevent erroneous detection of a head part of the golf club in the detected golf club,
A golf swing analysis method using multiple cameras and a single inertial sensor.

Images