JumpMetric: Assessment of Fiducial Positions for Vertical Jump Height Estimation from Depth Cameras and Wearable Sensors

Initially released for gaming applications, commercial, off-the-shelf depth cameras, such as the Microsoft Azure Kinect used in this research, have proven themselves useful for tracking the users’ body skeletal model and analyzing their motion in real time. Since the sensor observes the user from an external perspective, it requires a free line of sight and a suitable view angle to track the body parts without occlusion [16, 17, 27, 30]. However, it was demonstrated that even partially occluded body parts can be modeled and simulated for signal reconstruction [18].

Previous research evaluated the agreement of depth cameras and marker-based motion capturing systems, the gold standard, for the monitoring of “functional movements”, such as squats [17]. The results showed a moderate to high validity and that, in general, the impact of the view angle was insignificant. However, the achieved accuracy decreased significantly for occluded body parts, e.g., a hidden leg in the side view. Similarly, the joint angles have successfully been estimated during diverse physical activities, including vertical jumps [7, 26]. With a focus on the take-off phase, the trajectories of knee and shoulder joints during vertical jumping have been tracked with a depth camera and compared to those of a conventional camera-based 2d motion analysis system [15]. With a focus on the landing phase, instead, other research found the joint angles “relatively consistent” between a depth camera and a camera-based motion analysis system [11]. While most research focuses on ergonomics, injury prevention, and rehabilitation, the actual vertical jump height, estimated from depth camera data, has to our knowledge never been in the focus of interest yet.

In general, accelerometers have become very popular in the field of human activity recognition (HAR) [2]. The highly integrated sensors measure the rate of change in velocity of a body in its own instantaneous rest frame. Termed as gravity, a resting body on the Earth’s surface experiences an acceleration of 1 g or about 9.81 m/ s²  toward the Earth’s center of mass. Therefore, jumping causes an acceleration directed contrary to the force of gravity, which makes it possible to estimate the vertical jump height from accelerometer readings. Attached to the hip, lower back, and waist level, accelerometers have already successfully been used to estimate vertical jump height [5, 6, 14, 23, 31]. According to Dias et al. [10], there are primarily two approaches: the double integration of the vertical reaction force (DIF), which can also utilize the acceleration and neglect the person’s mass, and the determination of the flight time (FT) to estimate the reached jump height, according to Kibele [19] and Moir [21].

On the one hand, since the DIF method calculates the displacement of the center of mass (COM), it is important that the subjects start a jump in the same position as they land in. On the other hand, the FT method relies on the determination of the time difference between take-off and landing. Since this time is even squared, as apparent from equation (2), it is very sensitive to inaccuracies in the measurements. The accurate and reliable identification of take-off and landing is, therefore, critical to obtain precise timestamps. Typically, the absolute value of acceleration rapidly increases at take-off, which allows to determine the point in time when an athlete leaves the ground. Similarly, it increases again for landing and usually exceeds the previous peak of the take-off. In previous research, the FT method showed a slight offset compared to professional equipment, but consistent estimates within the system suggest that it can still be useful for tracking an athlete’s individual progress and improvement over time [5, 21].

The vertical jump height has previously been estimated with only a single accelerometer placed at the subjects’ hips [5, 8]. According to Casartelli et al. [5], the estimation of the jump height using the FT method overestimated the jump height with a systematic error of about 70 mm but showed a relatively good precision of ±  27 mm. In contrast, the DIF method was less reliable, showed a “poor validity”, and a low precision with a standard deviation of more than ±  120 mm. Similarly, Conceição et al. [8] describe that the estimation of the jump height with the FT method resulted in an accuracy of 6.1  ±  17.1 mm while the DIF method showed a low precision of ±  122.7 mm.

Additionally, gyroscope sensors have been used to correct the sensors’ rotation during the jump [14, 23]. In comparison to stereophotogrammetry measurements, Picerno et al. [23] evaluated two IMU-based methods that achieved a significantly biased accuracy of − 127  ±  68 mm (r  =  0.31) without and marginally biased 6  ±  55 mm (r  =  0.87) by applying a gyroscope-based compensation method. Similarly, Heredia-Jimenez and Orantes-Gonzalez [14] used a force place as reference, which is “considered the gold standard method”, and achieved even better accuracy of 3  ±  33 mm.

In another approach, Althouse [3] investigated the validity of seven IMU sensor locations and evaluated 15 different models that weighted the sensors’ contributions according to the mass of the limbs and combined their estimates into one. With a root mean square error (RSME) of 2.0 (1.2 m/ s² ) between the acceleration derived from the model and the force plate, the best performance was achieved with IMU signals from the trunk, thighs, and feet.

In general, various factors must be taken into account to find the most suitable and sensitive body positions for wearable sensors. According to Zeagler [33], the weight of the sensors should not affect the wearer or their balance. The sternum, lower back, and waist are the most suitable positions for tracking full-body movements. In a standing posture, the body’s COM is typically about 100 mm lower than the navel [9]. While being perceived as a comfortable wearing position, the waist is also very close to the COM [32], which allows for the best full-body acceleration estimates. Diverse sensors are available in commercial wearable devices such as wristbands or smartwatches, but they primarily capture limb movements that are independent of the body as a whole. Gemperle et al. [12] formulated guidelines for the placement of wearable devices which highlight that the sensors should be placed with as little skin, soft tissue, and muscle movement as possible to be both reliable and comfortable to wear. Furthermore, the social acceptance should be considered when selecting body positions for the attachment of wearable devices [4].

The commercially available, off-the-shelf Microsoft Azure Kinect depth camera is used to record the subjects’ limb and skeletal joint positions. It applies the time-of-flight principle, which involves emitting infrared light with a projector and measuring the time until the reflected light returns from the objects and environment. The time difference between these two events is then used to calculate the distance to an object [20]. The Kinect camera allows for the simultaneous tracking of a total of 32 skeletal joint positions at a frame rate of 30 fps (frames per second). Fig. 1 illustrates the obtained skeletal model for a subject performing a CMJ, which involves standing, squatting, and reaching the peak of the jump. The orientation of each skeletal joint is represented by a normalized unit quaternion, a complex and continuous representation of orientation in 3d-space that avoids gimbal lock [13]. The joint positions are recorded relative to the camera’s global reference frame and measured in millimeters (mm).

As detailed in equation (1) and visible in the exemplary time series of Fig. 6, the vertical displacement of the joints, and hence the jump height h_d, is calculated by taking the mean \(\overline{y}_{25\%}\) of the first \(25 \,\%\) of the y values as a baseline and subtracting it from the detected peak position y_max:

The commercially available M5Stack M5StickC Plus is used as a cost-efficient, off-the-shelf wearable device to monitor the wearer’s motion. Built around the popular ESP32 microcontroller, it is used to record the measurements from the onboard 3-axis accelerometer sensor InvenSense MPU6886. As illustrated in Fig. 2, a total of 10 devices are attached to specific body positions, which are the participants’ lower neck, chest (sternum), hips (left and right), thighs (left and right), ankles (left and right), and wrists (left and right). The sensor nodes are attached to these positions using 3d-printed mountings and elastic Velcro straps, on the clothing layer, as presented in Fig. 3a. Only the sensor at the lower neck was directly taped to the participants’ skin with skin-friendly, non-irritating tape. All devices were attached to the body with the accelerometers’ y-axis pointing up toward the ceiling.

The acceleration signals are used to identify the moments of take-off and landing. All 10 devices simultaneously record the three sensor channels at a sampling rate of 100 Hz and assign a specific timestamp to each data point. The participant’s flight time, originally termed as “time in air” by Moir [21], can be derived from the timestamps associated with take-off t₀ and landing t₁. In combination with the gravitational acceleration g of approximately 9.81 m/ s² , the flight time then allows the approximation of the vertical jump height h_w by applying the following equation (2) [19, 21]:

Ground truth information is essential to evaluate the measurements obtained from the two sensing modalities by comparing their values to a reliable and valid reference. Therefore, the subjects’ jumps are additionally recorded with a conventional Sony Alpha 6000 camera. As shown in Fig. 3b, every participant is provided with an orange tracking dot that is affixed to their hip. Placed to the side of the participants, a measuring tape enables the visual quantification of their hip displacement in the recorded video footage. In order to do that, first, a frame is selected from the period of the subject standing in a stable position and the tracking point’s position is read from the tape as \(P^{\prime }_{s}\). Then, a second frame is identified with the subject reaching the peak of the jump \(P^{\prime }_{j}\), the largest displacement in the video. The vertical jump height \(h^{\prime }_{g} = \overline{P^{\prime }_{s}P^{\prime }_{j}}\) is then determined by subtracting the standing position \(P^{\prime }_{s}\) from the peak position \(P^{\prime }_{j}\) as \(h^{\prime }_{g} = \left| P^{\prime }_{j} - P^{\prime }_{s} \right|\).

Unfortunately, the participants’ hip trajectory is closer to the camera than the measuring tape and not running in one line, which inevitably causes a non-negligible perspective displacement error in the manual readings. As illustrated in Fig. 4, the error, caused by the different distances of camera to hip d_cp and camera to measuring tape d_cm, is corrected by applying Thales’ theorem on proportionality, as provided in the following equation (3):Depending on the position of the camera P_c, the actual hip positions P_s of the standing and P_j of the jumping participant are projected to the measuring tape with the positions \(P^{\prime }_{s}\) and \(P^{\prime }_{j}\). The shorter the distance between the trajectory of the subject’s hip displacement \(\overline{P_{s}P_{j}}\) and the parallel intercept of the projection on the measuring tape \(\overline{P^{\prime }_{s}P^{\prime }_{j}}\), the smaller the perspective displacement error. In contrast, the larger the distance from the camera position P_c to the subject’s trajectory \(\overline{P_{s}P_{j}}\), the narrower the spanned angle α and the smaller the perspective displacement error. As presented in Fig. 5, the camera was consequently placed at a larger distance to the jump reference frame, with a distance of 5.65 m at which the measuring tape’s scale was still readable in the video frames.

The study was conducted according to the university’s ethics guidelines and did not require dedicated approval. A total of 44 subjects (33 male and 11 female) were recruited through word-of-mouth (13) and a lecture (31) that requires the students to participate in a user study of their choice. Therefore, aside from a single non-student, the subjects were primarily young undergraduate (36) and graduate (7) students in the age of 23.0  ±  2.2 years, with a height of 178.3  ±  9.0 cm and a weight of 73.2  ±  13.3 kg, resulting in a characteristic body mass index (BMI) of 22.9  ±  3.4. Before participating in the study, all subjects were asked about any medical contraindications and gave written consent that they could perform the required jumps without any risk of physical injury. Subjects were informed that they could terminate the study at any time without giving reasons and that quitting would not have any negative consequences, e.g. on the conveying lecture. To ensure minimal movement of the wearable sensors, participants were instructed to wear tight-fitting clothing. This also assured the least possible movement of the orange tracking dot (Fig. 3b). The subjects were asked to wear low-cut shoes that do not cover the ankles so that the devices can be attached correctly.

During the user study, each participant performed a total of 5 CMJs. Before the recording of these, the correct performance of CMJs were explained verbally and demonstrated by the instructor. The subjects were also given the chance to try it once to get feedback on their correct implementation. For each of the consecutively performed jumps, the subjects had a time window of 30 s and about one minute of rest between the individual jumps. Between jumps, the correct location of all body-worn sensing devices was checked. If a wearable device slipped out of place or did not send any data during the recording, the jump was repeated after the rest time. In case that a subject did not land within the predefined jump reference frame, the attempt was marked as invalid and repeated, as well. For the recording, the subjects were encouraged to jump as high as they could.

The user study took place during the semester break in a quiet part of the university building and without passing visitors. The room had a pleasant temperature of around \(20 \,\mathrm{ \mathrm{^{\circ }\mathrm{\mathrm{C}}}}\). The subjects were given a separate room to change clothes and were able to perform the jumps unobserved by third parties with the door closed.

The subjects were asked to perform the CMJ within a predefined 600 × 800 mm²  jump reference frame, which was marked with colored tape on the floor, as shown in Fig. 5. This way, it was ensured that the subjects’ body parts would not leave the depth camera’s field of view during the performed jumps.

The depth camera was placed at a distance of 2.90 m to the front of the subjects and in a horizontal orientation 1.40 m above the floor. The subjects wore 10 sensing devices at specific body positions. On the right-hand side of the jump frame, and thus of the participants, a measuring tape was spanned from the floor to the ceiling of the room. The video camera was placed on the left-hand side at a distance of 5.65 m from the jump reference frame and recorded the side view of the jumping subjects with the orange tracking dot placed on their hip facing toward the camera. As previously explained, the camera was placed at a large distance to minimize the perspective displacement error.

The software of the Microsoft Azure Kinect depth camera allows to automatically fit a skeleton model on the identified human body and the export of up to 32 body joints which serve as fiducial positions. Although the dataset contains the recordings of all joints available, the assessment covers only the subset of the 7 most relevant positions: neck, thoracic spine, pelvis, left and right ankle, as well as left and right wrist. The vertical jump height is then determined from the difference between the baseline and the peak of the jump. The baseline is determined by averaging the vertical position prior to the jump (i.e., \(25 \,\%\) of the recording), with the subject standing in a calm and stable pose. The peak of the jump is then simply identified as the maximum vertical displacement along the y-axis. Please note that due to the reversed internal coordinate system of the Microsoft Azure Kinect, the maximum displacement is here the minimum value.

Two exemplary time series of the vertical joint displacement are showcased in Fig. 6. For the recordings of a well-performed CMJ (6a ), it is apparent that the subject smoothly lowered their entire body in a squatting motion before jumping upward. The subject then performed the jump with their legs stretched out and hands resting on the hips. In contrast, for the recordings of an invalid CMJ (6b ), the subject had their knees pulled up during the jump, which makes the accurate jump height estimation from the ankle joints difficult. Moreover, the subject was swinging their arms excessively, which in turn made it significantly more difficult to determine the jump height from the wrists.

The ten wearable devices were simultaneously transmitting the sensor readings to a computer. In most attachment positions, a successive increase of the vertical acceleration is observable until the take-off takes place. At the moment of the jump, the acceleration shows a sudden change as the subjects leap into the air. At about half the way through the jump, the acceleration meets the zero line of 0 g as the body finds itself in free-falling motion. The moment the feet touch the ground for landing, the acceleration again increases abruptly, typically even more significant than at the take-off due to a much greater force being exerted on the body.

Fig. 7 showcases recordings from different sensor positions for (7a) a well-performed and (7b) an invalid CMJ. Based on these observations and the manual analysis of the time series, the characteristic of crossing the baseline of about 1 g was identified as a simple yet suitable feature to automatically determine take-off and landing in the time series, and thus to calculate FT and to estimate the vertical jump height.

For the lower neck, sternum, and hip position, only the signal in the y-direction (upward) has been considered. After an initial increase of the acceleration during the squatting motion, the acceleration drops below 1 g. This point marks the take-off of the subject jumping upward. The landing is reached when the signal again exceeds the baseline of about 1 g. At the ankle, a little dip within the initial rising of the signal is visible as the starting point of the jump. This is due to most participants raising their heels slightly before take-off. During the landing, two spikes can be observed in the signal. This is caused by participants landing on the ball of their foot. The landing is assumed to happen between the two spikes. For the thigh, the same features were used for the landing, but the features, as seen at the lower neck, sternum, and hip positions, were used to determine the take-off.

Because the wrists can move independently from the COM’s jumping trajectory, it was not always and reliably possible to accurately estimate the jump height with any simple feature similar to the described ones. Fig. 7b shows an example in which no clear features can be identified to determine the FT accurately.

Using the pelvis joint as a fiducial position to determine the jump height showed both the best accuracy and precision with a mean error of \(\overline{\varepsilon _{d}}\) = -15.8  ±  23.3 mm. The thoracic spine at the approximate height of the sternum and the neck resulted in comparable mean errors of \(\overline{\varepsilon _{d}}\) = -24.2  ±  35.1 mm and \(\overline{\varepsilon _{d}}\) = -24.2  ±  49.1 mm, respectively, with slightly better precision for the thoracic spine. For body parts that undergo more independent movement during the jump, in particular for extremities, the results are considerably worse. With a mean error of \(\overline{\varepsilon _{d}}\) = -30.8  ±  160.2 mm, the ankles show acceptable accuracy but poor precision. For the wrists with a mean error of \(\overline{\varepsilon _{d}}\) = -329.4  ±  341.5 mm, neither accuracy nor precision are acceptable. This is also reflected by the correlation with r² rapidly decreasing from 0.945 (r = 0.972, p  <  0.000001) for the pelvis to 0.080 (r = 0.284, p  <  0.000001) for the wrists. The Bland-Altman plots are provided in Fig. 8.

For the depth camera, it is important to choose joints whose trajectory resembles the COM’s displacement. As visible in Fig. 6a, although offset by some distance individual to the subject’s body, the trajectories of pelvis, thoracic spine, and neck are usually quite similar. If the jump is performed well, also the wrists’ trajectory can be similar to the pelvis’ one. Since the ankles do not lower during the squatting, there is no downslope visible, but the difference between baseline and peak can still represent the jump height well if the legs are stretched out properly. Unfortunately, several participants tucked in their feet mid-jump and therefore artificially increased their estimated jump height, as showcased in Fig. 6b, which in turn increases the error observed at the fiducial position.

The sensor positioned on the lower neck showed the best precision with a mean error of \(\overline{\varepsilon _{d}}\) = 18.8  ±  29.0 mm. Unexpectedly, the best accuracy was obtained from the ankles and thighs with a mean error of \(\overline{\varepsilon _{d}}\) = -4.8  ±  35.2 mm and \(\overline{\varepsilon _{d}}\) = 5.8  ±  42.5 mm, respectively. The sternum of the chest shows similar accuracy and precision with a mean error of \(\overline{\varepsilon _{d}}\) = -6.1  ±  41.1 mm. Although close to the COM, the hips showed a comparatively poor mean error of \(\overline{\varepsilon _{d}}\) = 19.6  ±  30.3 mm. As previously noticed for the depth camera, the wrists again show the worst performance with an unacceptable mean error of \(\overline{\varepsilon _{d}}\) = -431.1  ±  2685.9 mm that is caused by unrealistic jump height estimates from erroneously determined flight times. The good agreement with ground truth, except for the wrists, is also reflected by the correlation with r² ranging from 0.914 (r = 0.956, p  <  0.000001) for the neck to 0.826 (r = 0.909, p  <  0.000001) for the thighs. The wrists do not show any correlation with ground truth (r² = 0.004, r =  − 0.064, p = 0.220939). Fig. 9 shows the Bland-Altman plots.

The results align with Conceição et al. [8] who observed a mean error of 53 mm for accelerometer sensors attached to the hip level. In general, sensors placed at positions that have less soft tissue [33] tend to show better accuracy due to less independent movement of the sensors due to inertia. In particular the sensor on the lower neck, which was taped directly onto the skin of the subjects, had quasi no room for independent movement and hence showed the best performance. Because the quadriceps is a big muscle that is responsible for hip flexion and knee extension [25], which are both movements essential to jumping [28], the sensors placed on the thigh naturally have to undergo a lot of movement. This is apparent in both the lowest precision and r² (apart from the wrists), but also in the time series of the acceleration (Fig. 7b). A special case are the sensors placed on the participants’ limbs (ankles and wrists) as they can move relatively independently from the jump motion and the trajectory of the COM. Movements performed during the flight time, however, can be ignored as the FT method solely relies on the take-off and landing time. This means that the signals obtained from the ankles can still yield good results as no matter the movement of the individual subjects, they always have to start and finish their jumps with their feet on the ground. However, this is not true for the wrists as they do not necessarily need to be in a certain position at the start or end of a jump. Because of this initial uncertainty, it was not possible to yield any reliable jump height estimate from the wrist-worn sensors.

In general, obtaining ground truth measurements of vertical jumps is not easy. While the mechanical Vertec offers only a limited resolution of typically 12.7 mm, other solutions such as contact mats and force plates allow for finer measures but come again with other limitations and disadvantages. Therefore, we decided to record the participants’ countermovement jumps with a video camera and manually read the ground truth or reference information from a measuring tape in the background of the captured video footage. A semi-transparent overlay of a frame with the uncovered measuring tape allowed for an accurate reading. We also minimized the perspective displacement by increasing the distance of the camera for a narrower view and then corrected the remaining error by applying Thales’s theorem on proportionality. Nevertheless, the participants’ hips were not all of the same width and not necessarily aligned with the outer contour of the jump reference frame. However, the positional errors within the reference frame are marginal and, for a subject with a typical jump height of 400 mm, a hip displacement of 100 mm within the reference frame would result in an error of less than 6 mm. This error inevitably affects the accuracy of all fiducial positions but can be disregarded in consideration of the other sources of inaccuracies, such as the modalities themselves. Also the manual reading of the vertical displacement from the video frames is prone to inaccuracies due to the limited resolution of the measuring tape, the background being slightly blurred, and the very subjective interpretation of the tracking dot’s position. Moreover, although requested in the study announcement, the clothing of several participants was not as body-tight as desired and the tracking dot might slightly have moved due to the movement of the textile on the skin.

Although we refer to the reference measurements as ‘ground truth’ for a better understanding of the study design, this term is not very precise. The previously mentioned inaccuracies of the reference measurements indicate that the reliability of the values is not absolute. Therefore, we decided to use Bland-Altman plots which are common in this research field [23] and intended to analyze the agreement between two modalities with some inherent error.

Although the wearable motion sensors also provide an onboard gyroscope, only the accelerometer data was recorded and used in this study. In future experiments, the gyroscope signals might be helpful to correct the alignment of the sensors [14, 23] and to further improve the accuracy. Moreover, gyroscopes and the advanced analysis of the subjects’ motion sequences might even allow for obtaining accurate estimates from sensor positions that failed in this study.