[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
A Survey of the Magnetic Anisotropy Detection Technology of Ferromagnetic Materials Based on Magnetic Barkhausen Noise
Next Article in Special Issue
Myoelectric-Based Estimation of Vertical Ground Reaction Force During Unconstrained Walking by a Stacked One-Dimensional Convolutional Long Short-Term Memory Model
Previous Article in Journal
Kolmogorov–Arnold Network in the Fault Diagnosis of Oil-Immersed Power Transformers
Previous Article in Special Issue
Walk Longer! Using Wearable Inertial Sensors to Uncover Which Gait Aspects Should Be Treated to Increase Walking Endurance in People with Multiple Sclerosis
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Quantifying Arm and Leg Movements in 3-Month-Old Infants Using Pose Estimation: Proof of Concept

1
School of Health and Rehabilitation Sciences, The Ohio State University, Columbus, OH 43210, USA
2
Heart Center Nationwide Children’s Hospital, Columbus, OH 43210, USA
3
School of Health and Rehabilitation Sciences, College of Nursing, The Ohio State University, Columbus, OH 43210, USA
4
Abigail Wexner Research Institute Nationwide Children’s Hospital, Columbus, OH 43210, USA
*
Authors to whom correspondence should be addressed.
Sensors 2024, 24(23), 7586; https://doi.org/10.3390/s24237586
Submission received: 16 October 2024 / Revised: 23 November 2024 / Accepted: 26 November 2024 / Published: 27 November 2024
(This article belongs to the Collection Sensors for Gait, Human Movement Analysis, and Health Monitoring)

Abstract

:
Background: Pose estimation (PE) has the promise to measure pediatric movement from a video recording. The purpose of this study was to quantify the accuracy of a PE model to detect arm and leg movements in 3-month-old infants with and without (TD, for typical development) complex congenital heart disease (CCHD). Methods: Data from 12 3-month-old infants (N = 6 TD and N = 6 CCHD) were used to assess MediaPipe’s full-body model. Positive predictive value (PPV) and sensitivity assessed the model’s accuracy with behavioral coding. Results: Overall, 499 leg and arm movements were identified, and the model had a PPV of 85% and a sensitivity of 94%. The model’s PPV in TD was 84% and the sensitivity was 93%. The model’s PPV in CCHD was 87% and the sensitivity was 98%. Movements per hour ranged from 399 to 4211 for legs and 236 to 3767 for arms for all participants, similar ranges to the literature on wearables. No group differences were detected. Conclusions: There is a strong promise for PE and models to describe infant movements with accessible and affordable resources—like a cell phone and curated video repositories. These models can be used to further improve developmental assessments of limb function, movement, and changes over time.

1. Introduction

Advancements in technology, digital health, and remote monitoring are making the ability to collect sophisticated and objective pediatric movement data feasible, cost-effective, and ecological [1,2]. Devices such as wearable sensors [1,2,3] and portable markerless multi-camera 3D motion analysis systems [4,5,6,7,8] make the collection of human movement data possible in many environments (i.e., home, day care, sports field, research laboratory). The potential of these systems to detect, identify, and measure changes in symptoms, including those that are present in everyday life, is exciting. While these tools produce meaningful results, they require either a device attached to the infant (wearable) or specialized cameras. However, there is another analysis tool that can address these limitations and has shown promise in measuring movements in adults and children from a video recording [9].
This emerging tool that has shown promise for analyzing movements from simple video recordings is pose estimation [6,7,8,9]. Pose estimation can track movement in a video by recording the pixel’s position over time. The pixels from the video measurement can be converted to an x–y–z coordinate for an estimate of position. Depth, or the z-direction, cannot be calculated with a single camera to our knowledge. Cameras project a scene onto a 2D space and data are lost if motion extends outside of the limited field of view. Pose estimation has been used to measure 2D movements of the legs in athletes [10], adults with stroke [9,11,12], and patients with Parkinson’s Disease [9]. Pose estimation has also been used to assess the arm of adults with stroke [9] and the fingers in patients with Parkinson’s Disease [13]. Unlike wearable sensors and markerless motion capture with multiple cameras, pose estimation does not require the participant to wear additional equipment and can be used from a single camera recording for movement primarily in a single plane. This is particularly compelling for infants who are wearing several pieces of medical equipment in intensive care units or who do not want to wear markers for 3D motion analysis [4,5,9]. The potential is high because it means movement and biomechanical data can be collected and analyzed across many environments, with minimal training, at a cost-effective price, part of in-person or remote digital monitoring, and for use with retrospective video repositories and curated clinical data sets [14,15,16,17].
While pose estimation has the promise to quantify pediatric movement from a video recording, firstly, preliminary validation of data processing techniques is required to develop robust measures of infant movement. Most pose estimation models have been developed on adult populations. Infant bodies’ parameters are often difficult to identify with models developed on adult proportions given the anthropometric proportions and size differences (from adults) [18,19]. Additionally, movement type varies greatly across human development, and the methods used to assess adult movement might not be appropriate for pediatric movement. Therefore, the currently available models for pose estimation must be tested and validated for infant use.
Currently, in the pose estimation literature, pediatric movement has been quantified using pixels as the unit of measurement (e.g., pixels per second for velocity, or a change in pixels for displacement) for spontaneous movement of the arms and legs [8,9]. These pixel measurements [6,20] are then correlated with clinical or research tools like the general movement assessment (GMA) or diagnosis of developmental disabilities as an early measure of concurrent or predictive validity. These studies show the potential that pose estimation can (1) estimate and localize body postures with a single camera view accurately, (2) quantify infant movements, (3) be used with automated computer-based methodologies, and (4) have early validity with clinical measures. However, pose estimation can extract meaningful infant behavioral analytics that are easier than pixels to understand for research and clinical practice related to movements [21]. For example, variables such as the frequency counts of movements (i.e., the number of movements or quantity of movements) and how often similar movements are repeating themselves (i.e., the sample entropy or complexity) can be calculated.
Quantity [2,22,23] and complexity [22,24] of movement are powerful measures in the child developmental and rehabilitation literature, because they are correlated with motor skill development and are being developed as a tool for gauging motor rehabilitation of children with developmental disabilities [25,26]. Therefore, there is a need to validate a way to process pose estimation data using units that are commonly used in order to compare and integrate within the current literature base. Quantifying the acceleration of movements in the limbs, similar to how these measures are calculated with wearables, will allow us to use pose estimation to quantify the number of movements and the complexity of movement in infant populations.
The purposes of this study were to (1) test the accuracy of an open-source PE model, called MediaPipe, to detect arm and leg movements in 3-month-old infants with and without complex congenital heart disease (CCHD); (2) compare the frequency and complexity of arm and leg movements between infants with and without CCHD; and (3) establish preliminary concurrent validity with the Test of Infant Motor Performance and the Bayley Scale for Infant and Toddler Development.
Retrospective data from infants with and without CCHD were selected for this proof of concept to assess pose estimation in a clinical population and a comparison group. Infants with CCHD are at risk of neurodevelopmental disabilities and video assessment using pose estimation of spontaneous movements and no manipulation of a sick infant in the pediatric intensive care units might benefit this population. In addition, inclusion of both typical development and CCHD populations were included to make sure there was a sufficient range and variability of spontaneous behaviors.

2. Materials and Methods

2.1. Participants

A subset of video data from a larger study [27] was used for this study. Participants were full-term infants (>36 weeks gestational age), 3 months of age, with (n = 6) and without (n = 6) complex congenital heart disease (CCHD). All the infants with CCHD required surgical intervention within the first months of life.

2.2. Procedures

Participants were assessed in their homes using the Test of Infant Motor Performance (TIMP) [28], the Bayley Scale for Infant and Toddler Development (3rd edition) [29], and a 15 min contingency mobile paradigm. In the contingency paradigm, movements of the right leg activate the mobile. Video data from the contingency paradigm were used in this study and were recorded by a Sony 8mm CCD-TRV608 at 30 frames per second. The whole 15 min video was processed using the PE model.
Videos were selected if, for the entire video, the infant was in full view, in the frontal plane, and the mobile did not come into the camera view. This resulted in videos that could view the baby for the whole paradigm and did not have time points where external entities (i.e., the mobile, a parent, or an assessor) blocked the view of the child, which might interfere with locating virtual anatomical markers. The video repository contained 46 videos (35 infants with CCHD and 11 infants with typical development (TD)) and 12 were used. Five videos of infants with TD were excluded for occlusions of the infant (i.e., the infant was not in full camera view for the entire video), yielding six videos of infants with TD. The video files of infants with CCHD were randomly run through the model until a balanced group was acquired. Thirteen video files were needed to acquire a sample of six video files from the infants with CCHD. Seven video files from the infants with CCHD were excluded for occlusions of the infant.

2.3. Data Processing

Video data (30 frames per second) were imported and processed using custom python (version 3.9) and MATLAB (R2023b) scripts. Python (version 3.9) was used to extract the position in the x and y coordinates for 33 anatomical markers (referred to as virtual markers) from the MediaPipe model (Figure 1) [30]. MATLAB (2023b) was used to quantify the acceleration of arm and leg movement using the position of the virtual markers of the ankle and wrist. Arm and leg movements are the focus of this manuscript.
Prior to MATLAB processing, output videos were visually inspected to determine that the pose estimation model was correctly placing the virtual markers on the correct anatomical locations of an infant. Unrealistic values for virtual markers can be caused by items and people that block the view of the child, such as the rail of a crib, a caregiver’s hand, or part of the mobile coming into camera view. Obstructions that only occurred for a second of the video data (i.e., an ID card for labeling the video data) or had infrequent occurrences (i.e., a caregiver manipulating a pacifier for 1 s 2 times during the collection) were included in the data set. Video data with obstructions that occurred for longer than 10 s (a crib rail, obstructing the view of a child) or frequently occurred (i.e., a mobile consistently coming into view) were excluded. Infant behaviors like bringing hands to midline, placing hands in the mouth, and grasping feet did not interfere with the model’s trace of the infant and were included in the data set.
Since MediaPipe exports the virtual markers in pixels, we converted pixels to meters using the individual x and y coordinates for both virtual markers of the shoulder, a standardized norm value for shoulder-to-shoulder distance (S2S) for 3-month-old infants (0.18415 m) [31], and trigonometry. The distance between the virtual shoulder markers was calculated by taking the arc tangent of the absolute difference between the y coordinates divided by the absolute difference between the x coordinates (Equation (1)). This was performed for each frame of data, resulting in the shoulder-to-shoulder distance in pixels for every frame. The median value for shoulder-to-shoulder distance was quantified for the whole file and then was divided by 0.18415 m (the norm value for shoulder-to-shoulder distance in 3-month-old infants) (Equation (2)). This resulted in the conversion of meters to pixels and was used to convert pixels to meters for all virtual markers in each participant’s data set. These data were smoothed using the Gaussian filter in MATLAB with a window frame of 8.
S 2 S = tan 1 y l i y r i x l i x r i
C o n v e r s i o n = M e d i a n   S 2 S 0.18415
Position data from the virtual ankle and wrist markers and time were used to quantify acceleration of the limbs (i.e., the change in velocity over time in x and y). Infant leg and arm movements were quantified using the peak movement acceleration of the ankle and wrist virtual markers for both sides. This was adapted from Smith et al. (2015) [2]. Acceleration was quantified using the square root for the sum of squares from the acceleration in the x and y coordinate of each leg and arm (see Formula (3)). Acceleration peaks were identified using the findpeaks function in MATLAB. Each peak was identified using the video timeline that was exported from the Python (version 3.9) output using the MediaPipe model (see Figure 2). Arm and leg movements were defined as an acceleration peak greater than 3 m/s2 and occurring in more than 10 frames of video data (i.e., 0.33 s) from the prior movement of the corresponding limb. The acceleration threshold was chosen based on Fitter et al. (2019) [32] and the time threshold of 10 frames of video data (i.e., 0.33 s) was chosen based on piloting.
a c c e l e r a t i o n = x 2   +   y 2
Leg and arm movement rates per hour were calculated by dividing the number of movements produced in the whole 15 min video and the duration of the video in minutes. Then, the results were multiplied by 60 min to convert the value to movements per hour. This unit was chosen to more easily compare the results from pose estimation data to previously reported values in the wearable sensor literature, which reports movements per hour [2,25].
Using the acceleration signal from each limb, sample entropy for each limb was calculated in MATLAB (2023b) using a custom program. This program has been used in Deng et al. (2022) [22] and Smith et al. (2017) [24] and was developed based off Richman and Moormman (2000) [33].

2.4. Behavioral Coding of Leg and Arm Movements

Firstly, behavioral coding was used as the gold standard and identified leg and arm movement for 20 s for each video as a measure of validity [2]. A movement expert with doctoral training identified the onset and offset of a leg or arm movement. A leg movement was operationally defined as a visible change in the ankle’s location (i.e., displacement) with flexion or extension of the hip and knee. An arm movement was operationally defined as a visible displacement of the wrist with joint movement at the shoulder or elbow. The timing of movements from the annotated video file was compared with results from the MATLAB code and the agreements and disagreements between the data were identified (Table 1).

2.5. Measurements

The Bayley Scale for Infant and Toddler Development [29] is an assessment that measures a child’s cognitive, language, and motor development. Children perform a set of items for each domain until they score a 0 on 5 consecutive items. This assessment is used in children 0–42 months of age. Bayley scores for the cognitive, communication, and motor domains were used for this analysis.
The Test of Infant Motor Performance (TIMP) [28] is an assessment of an infant’s motor development. It is used in infants 34 weeks post-conception to 4 months post-term and was developed to identify infants with delayed motor development and to track the progress of children with typical development.

2.6. Statistics

All statistics were performed in MATLAB (2023b). The positive predictive value (i.e., the likelihood of identifying a movement correctly) and sensitivity (i.e., it is the likelihood that a movement is identified, if a movement is known to have occurred) were used to assess the accuracy of the model. The positive predictive value was calculated by dividing the total number of agreed movements identified between behavioral coding and pose estimation by the sum of movement identified by both methods. Sensitivity was calculated by dividing the total number of agreed movements identified by both methods by the total number of movements identified using behavioral coding.
Wilcoxon rank sum tests were used to determine differences between CCHD and infants with typical development (TD) for the positive predictive value, limb movement rate, sample entropy, and Bayley scores and TIMP scores. Pearson correlations were used to determine effect sizes for the comparison of CCHD and TD for the following variables: average leg and arm movement rate (the average of left and right limb) and sample entropy for all limbs. Pearson r can be explained using the following values for r: small effect was |r| > 0.1, moderate effect |r| > 0.3, and large effect |r| > 0.5 [34]. Spearman correlations were used to examine the association between the average leg and arm movement rate, sample entropy, and Bayley and TIMP scores for concurrent validity. Pearson r was also used to assess the effect size between the movement rate of the arms and legs, with the Bayley and TIMP scores.

3. Results

3.1. Participants

The median age for TD was 3.54 months (range: 3.4–3.9) and CCHD was 3.4 months (range: 3.06–3.73). A total of 75% were male (n = 3 female) and racial composition of the participants was 83% White and 16% Black. None of the children were Hispanic.

3.2. Model’s Accuracy to Detect Leg and Arm Movement in TD and CCHD

We identified 499 leg and arm movements, and the pose estimation model had a positive predictive value of 85% and a sensitivity 94% (Table 1a). The pose estimation model’s positive predictive value in TD was 84% and sensitivity was 93% (Table 1b). The pose estimation model’s positive predictive value in CCHD was 87% and the sensitivity was 98%. (Table 1c). No significant differences were found between the groups for the pose estimation model’s positive predictive value (p > 0.05).
Table 1. Two-by-two contingency tables comparing the agreement and disagreements between the detection of limb movements using the pose estimation and behavioral coding for the following: (a) the combination of all children, (b) children with typical development, and (c) children with complex congenital heart disease. The section with no movement (i.e., the cell with “No behavioral coded movement” and “No pose estimation Movement”) is left blank because it is not possible to calculate the number of movements that did not happen.
Table 1. Two-by-two contingency tables comparing the agreement and disagreements between the detection of limb movements using the pose estimation and behavioral coding for the following: (a) the combination of all children, (b) children with typical development, and (c) children with complex congenital heart disease. The section with no movement (i.e., the cell with “No behavioral coded movement” and “No pose estimation Movement”) is left blank because it is not possible to calculate the number of movements that did not happen.
(a)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement49957
No pose estimation movement30
(b)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement34537
No pose estimation movement27
(c)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement15420
No pose estimation movement3
When examining leg and arm movements separately, overall, the positive predictive value was 84% in legs (Table 2a) and 86% in arms (Table 3a). Overall, the sensitivity was 94% in legs (Table 2a) and 95% in arms (Table 3a). For TD, the positive predictive value was 83% in legs (Table 2b) and 86% in arms (Table 3b), while the sensitivity was 92% in legs (Table 2b) and 94% in arms (Table 3b). For CCHD, the positive predictive value was 87% in legs (Table 2c) and 87% in arms (Table 3c), while the sensitivity was 99% in legs (Table 2c) and 97% in arms (Table 3c).

3.3. Differences in Average Limb Movement Rate

For individual infants using movements/hour, movement rates ranged from 399 to 4211 movements per hour for legs and 236 to 3767 movements per hour for arms. The median movement rate for TD was 2357 movements per hour for legs (range: 1327–4211 movements per hour) and 1373 movements per hour for arms (range: 912–3767 movements per hour). The median movement rate for CCHD was 1199 movements per hour for legs (range: 399–4155 movements per hour) and 1231 movements per hour for arms (range: 236–3764 movements per hour). TD tended to produce more movements than infants with CCDH; however, no significant differences were found for the hourly leg and arm movement rates (Figure 3). When comparing groups, there was a medium effect size for the leg rate (r = 0.34) and a small effect size for the arm rate (r = 0.25). This suggests that compared to CCHD, the leg rate and arm rate are higher in the TD group.

3.4. Sample Entropy of the Acceleration Signal for Arm and Leg Movements

Median sample entropy in the left and right arms was 0.27 and 0.32 for CCHD, respectively, and 0.29 and 0.27 for TD. Median sample entropy in the left and right legs was 0.25 and 0.32 for CCHD, respectively, and 0.37 and 0.41 for TD. The box plot in Figure 4 depicts the sample entropy for both groups in all four limbs. There was no statistical difference between the groups for sample entropy for left and right arm and leg movement (p > 0.05). Infants with TD tended to have larger sample entropies for both side and limbs, meaning that the TD group may have produced more variability in their movements. There was a medium effect size for left leg movements (r = 0.32) and a low effect size for left (r = 0.25) and right (r = 0.13) arm movements and right leg movements (r = 0.15), suggesting that the TD group showed more variability in the leg that activated the mobile (i.e., the right leg).

3.5. Associations with Movement Rate and Developmental Assessments

Bayley communication was significantly correlated with the arm movement rate (p = 0.04, r = 0.6). No other associations were found between the leg and arm movement rate and the TIMP and the motor and cognitive scale (p > 0.05) (see Table 4). However, Pearson’s correlations showed medium effect sizes between TIMP scores and the leg (r = 0.30) and arm (r = 0.38) movement rate, as well as Bayley communication scores and arm movement rate (r = 0.35). This suggests that infants with higher TIMP scores had higher leg and arm movement rates, and those with higher Bayley communication scores had higher arm movement rates. Small effect sizes were found between Bayley cognitive scores and arm movement rate (r = 0.25), and Bayley communication scores and leg movement rate (r = 0.23). This suggests that infants with higher Bayley cognitive scores had higher arm movement rates, and those with higher Bayley communication scores had higher leg movement rates.

4. Discussion

This paper preliminarily validates a data processing technique to quantify and describe infant limb movement. The two groups were compared to (1) validate these measures in a clinical population as well as a population with typical development (TD), and (2) perform an exploratory comparison that will lead to further robust questions using pose estimation. The methodology presented in this study shows that there is considerable promise for quantifying movement in 3-month-old infants using pose estimation. Firstly, there was high PVV (i.e., all values above 80%) and sensitivity (i.e., all values about 90%) with behavioral coding. Secondly, using the model alone, the ranges of arm and leg movements are similar to those reported in the wearables literature. Thirdly, although nonsignificant hypothesis testing was performed to compare CCHD to TD, the following reasons support the trends and effect sizes observed: (1) the right leg showed higher values as would be expected in the mobile paradigm with a tethered right leg, (2) the CCHD infants had lower mean rates as might be expected by an infant population that spent considerable time in the PICU and had major heart surgery, and (3) the complexity measured by sample entropy is similar to those values reported in the literature for infant movements using other methods.
By developing a method for integrating a familiar measurement into the system, pixels can be converted to meters and be used to measure movement in standardized ways that are easily understood by clinicians and researchers. Additionally, the data processing techniques described in the study show that we can also quantify the complexity of movement that the infants are producing through measurement of the sample entropy. Therefore, it is possible to quantify the amount of movement and its complexity through video data. The potential to analyze movements from any video recording, through digital health appointments, during clinical assessments, and with any video repository may add a layer of precision and prediction that was once only possible in a laboratory setting.

4.1. Frequency

Quantifying common variables used in the wearable sensor literature and applying it specifically to pose estimation data with pixels provides a way to study infant movement that is translatable to other areas of the biomechanical literature. The movement rate values in the two groups were very similar to those that have been previously reported in the wearable sensor literature and behavioral coding literature. The hourly leg movement rate has ranged from 813 to 3671 in infants with TD [2], and in recent studies, it has been found to be higher [26]. For the arm movement rate data, 3-month-old infants perform about 2011 movements per hour [35]. These values are similar to the data reported in this paper (leg rate range: 399–4211 movements per hour; arm rate range: 236–3767 movements per hour) (see Figure 3).
In addition to the movement rate values being similar to values reported in the wearable sensor literature, the trend for infants with TD to produce more movements than CCHD is represented in the literature. Children at risk of neurodevelopmental disabilities, including CCHD, have been found to produce fewer movements compared to those with TD [25,36,37,38]. It is for this reason that early interventions sometimes focus on increasing the movement rate and motor experience as soon as possible [38,39,40]. The results from this paper offer a measurement that could be implemented in a variety of settings to help gauge the amount of movement infants at risk of neurodevelopmental disabilities are producing. Additionally, future work could explore the possibility of using pose estimation in combination with other tools as a biomarker to recommend early intervention and treatment response.
The sample entropy values of arm and leg movements in our sample were slightly higher than those reported in the literature, but still within the range of the reported values [22,24]. The movement characteristics of sample entropy are difficult to compare between studies (e.g., different movements, tasks, and participant characteristics) because the interpretation of sample entropy is context-specific. For a healthy adult walking on a treadmill, you would expect a low sample entropy, because their movements will be consistent and less variable, while an infant learning to walk would have a higher sample entropy, due to more variability in their movements. Therefore, comparisons between the sample entropy found in this study with past studies should be examined with caution.
While wearable sensors can be used to collect full data recordings and can account for movements that are the results of rotation [2,41], pose estimation data can be used to study concepts that are challenging using wearable sensor methodology. For example, full-day wearable sensor data do not describe the context in which movement is occurring unless it is paired with written diaries (e.g., activity logs) or surveys (e.g., ecological momentary assessment) [1]. Pose estimation offers the ability to look at a recording of the movement and further analyze it once the data have been processed. This offers the potential to study sensorimotor experience and how infants produce motor behaviors in response to different stimuli like touch from a caregiver or themselves, voice, and light. For these studies, pose estimation may support or make behavioral coding efficient for movement variables, making data processing much quicker and more feasible with larger samples.

4.2. Two-Dimensional vs. Three-Dimensional Motion Analysis

In comparison to markered 3D motion analysis [42], pose estimation does not require an infant to wear additional equipment like markers, and thus can be performed at a much lower cost [9]. This is very promising because pose estimation could be used in infant populations (or any participants) that do not tolerate markers or pull off markers when they are placed on landmarks. While pose estimation is limited by the fact that all the anatomical landmarks in the model need to be present on the participant in the video recording, future work can focus on creating new pose estimation models that can analyze infant movement in the prone position, when a specific body part is occluded due to therapies like constrain-induced movement therapy [43], or in a specific body position like prone or sitting.

4.3. Preliminary Findings Associating Movement Rate and Clinical Assessments

While our sample size lacked the power to detect associations between the movement rate and clinical assessment data, the results suggest statistical trends that inform further study. We found several medium effects when looking at the association between the leg and arm movement rate and the TIMP, as well as the Bayley communication score and arm movement rate. This suggests possible associations between movement captured using pose estimation data and clinical tools used to evaluate and gauge a patient’s motor improvements during and after a treatment. These associations should be explored further with a larger sample in the attempts to look at future biomarkers that could identify developmental disabilities at earlier ages.

4.4. Limitations

The sample size in this study was determined by the amount of data available according to the specific parameters set to obtain an accurate trace of the infant’s movement. This may be a contributing factor for not finding a significant difference between the two groups. However, our effect sizes show promising results and suggest that these data can be used to determine movement differences in these two populations with a larger sample size. Future studies with a larger sample could examine movement differences in different pediatric populations (i.e., children at risk of neurodevelopmental delays) that may help identify predictors of optimal developmental outcomes. One direction is to use pose estimation in the neonatal intensive care unit. Pose estimation could be used to examine strategies to promote optimal neuromotor outcomes and to better detect developmental disabilities such as cerebral palsy.
Another limitation is that the amount of data from the video files could have been maximized using video trimming or cropping the video file to focus on the region of interest. However, these data were not collected for pose estimation and were used respectively. Future work should aim to present the child in the camera view for the full length of the video, or design projects that allow for segmenting data into blocks.
The sampling rate for the video data was 30 frames per second. Chambers et al. (2020) [8] suggest that pose estimation data should be collected at the highest sampling rate possible with appropriate lighting to improve accuracy. Higher sampling rates are possible given the rapid advancement in the accessibility of video camera data. In fact, most Apple products (e.g., iPhone and iPad) can record video data at 60 frames per second. While higher sampling rates will most likely result in more accurate data, future study is needed to examine how much more accurate these data are and if the increase in accuracy outweighs the cost for storing twice as much data.

5. Conclusions

These novel data and analysis show that we can quantify the amount and complexity of infant limb movements using pose estimation. Since pose estimation only requires a single video camera, future studies aimed at understanding the neural correlates of infant movement could be conducted in lower-income areas of the world, similar to the wearables sensor [44] and smartphone literature [1,45], and in situations where devices like wearable sensors cannot be placed on a child. Thus, pose estimation opens new and inexpensive avenues for the analysis of infant movement and the early detection of developmental disabilities.

Author Contributions

Conceptualization, M.R.R. and J.H.; methodology, M.R.R. and J.H.; software, M.R.R.; validation, M.R.R. and J.H.; formal analysis, M.R.R.; investigation, M.R.R., J.S., T.K. and J.H.; resources, J.S., T.K. and J.H.; data curation, M.R.R. and J.H.; writing—original draft preparation, M.R.R. and J.H.; writing—review and editing, M.R.R., J.S., T.K. and J.H.; visualization, M.R.R.; supervision, J.H.; project administration, J.H.; funding acquisition, M.R.R. and J.H. All authors have read and agreed to the published version of the manuscript.

Funding

These data were supported from a research grant from the American Heart Association, GRNT27760156; and C-Progress, grant number: 5P2CHD101912-05, PI: Ramey. Dr. Rosales’s work on this paper was sponsored by The Ohio State University President’s Postdoctoral Scholarship award.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of Nationwide Children’s Hospital (IRB# IRB13-00375/Ame00011479, date of approval: 7/12/2016).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the caregivers of the patient(s) to publish this paper.

Data Availability Statement

Data will be made available upon request from the corresponding author.

Acknowledgments

The authors would like to thank the infants and their families for their participation in this study. Additionally, we would like to thank the researchers who collected these data.

Conflicts of Interest

The authors declare no conflicts of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

  1. Rosales, M.R.; Rohloff, P.; Vanderbilt, D.L.; Tripathi, T.; Valentini, N.C.; Dusing, S.C.; Smith, B.A. Collecting infant environmental and experiential data using smartphone surveys. Pediatr. Phys. Ther. 2021, 33, 47–49. [Google Scholar] [CrossRef]
  2. Smith, B.A.; Trujillo-Priego, I.A.; Lane, C.J.; Finley, J.M.; Horak, F.B. Daily quantity of infant leg movement: Wearable sensor algorithm and relationship to walking onset. Sensors 2015, 15, 19006–19020. [Google Scholar] [CrossRef]
  3. Deng, W.; Nishiyori, R.; Vanderbilt, D.L.; Smith, B.A. How Many Days are Necessary to Represent Typical Daily Leg Movement Behavior for Infants at Risk of Developmental Disabilities? Sensors 2020, 20, 5344. [Google Scholar] [CrossRef] [PubMed]
  4. Krasowicz, K.; Michoński, J.; Liberadzki, P.; Sitnik, R. Monitoring Improvement in Infantile Cerebral Palsy Patients Using the 4DBODY System—A Preliminary Study. Sensors 2020, 20, 3232. [Google Scholar] [CrossRef] [PubMed]
  5. Schroeder, A.S.; Hesse, N.; Weinberger, R.; Tacke, U.; Gerstl, L.; Hilgendorff, A.; Heinen, F.; Arens, M.; Dijkstra, L.J.; Pujades Rocamora, S.; et al. General Movement Assessment from videos of computed 3D infant body models is equally effective compared to conventional RGB video rating. Early Hum. Dev. 2020, 144, 104967. [Google Scholar] [CrossRef]
  6. Lin, S.C.; Chandra, E.; Tsao, P.N.; Liao, W.C.; Chen, W.J.; Yen, T.A.; Hsu, J.Y.J.; Jeng, S.F. Application of Artificial Intelligence in Infant Movement Classification: A Reliability and Validity Study in Infants Who Were Full-Term and Preterm. Phys. Ther. 2024, 104, pzad176. [Google Scholar] [CrossRef] [PubMed]
  7. Cherry-Allen, K.M.; French, M.A.; Stenum, J.; Xu, J.; Roemmich, R.T. Opportunities for Improving Motor Assessment and Rehabilitation after Stroke by Leveraging Video-Based Pose Estimation. Am. J. Phys. Med. Rehabil. 2023, 102, S68–S74. [Google Scholar] [CrossRef]
  8. Chambers, C.; Seethapathi, N.; Saluja, R.; Loeb, H.; Pierce, S.R.; Bogen, D.K.; Prosser, L.; Johnson, M.J.; Kording, K.P. Computer Vision to Automatically Assess Infant Neuromotor Risk. IEEE Trans. Neural Syst. Rehabil. Eng. 2020, 28, 2431–2442. [Google Scholar] [CrossRef]
  9. Lam, W.W.T.; Tang, Y.M.; Fong, K.N.K. A systematic review of the applications of markerless motion capture (MMC) technology for clinical measurement in rehabilitation. J. Neuroeng. Rehabil. 2023, 20, 57. [Google Scholar] [CrossRef] [PubMed]
  10. Baumgartner, T.; Paassen, B.; Klatt, S. Extracting spatial knowledge from track and field broadcasts for monocular 3D human pose estimation. Sci. Rep. 2023, 13, 14031. [Google Scholar] [CrossRef] [PubMed]
  11. John, K.; Stenum, J.; Chiang, C.C.; French, M.A.; Kim, C.; Manor, J.; Statton, M.A.; Cherry-Allen, K.M.; Roemmich, R.T. Accuracy of Video-Based Gait Analysis Using Pose Estimation During Treadmill Walking Versus Overground Walking in Persons After Stroke. Phys. Ther. 2024, 104, pzad121. [Google Scholar] [CrossRef]
  12. Aravamuthan, B.R.; Pearson, T.S.; Ueda, K.; Miao, H.; Zerafati-Jahromi, G.; Gilbert, L.; Comella, C.; Perlmutter, J.S. Determinants of gait dystonia severity in cerebral palsy. Dev. Med. Child Neurol. 2023, 65, 968–977. [Google Scholar] [CrossRef] [PubMed]
  13. Spann, J.; Chen, S.A.; Ashizawa, T.; Hoque, E. Getting on the Right Foot: Using Observational and Quantitative Methods to Evaluate Movement Disorders. In Proceedings of the 29th International Conference on Intelligent User Interfaces, Greenville, SC, USA, 18–21 March 2024; pp. 742–749. [Google Scholar] [CrossRef]
  14. Dressler, V. Databrary. Ref. Rev. 2015, 29, 32–39. [Google Scholar] [CrossRef]
  15. Nieuwenhuys, A.; Õunpuu, S.; Van Campenhout, A.; Theologis, T.; De Cat, J.; Stout, J.; Molenaers, G.; De Laet, T.; Desloovere, K. Identification of joint patterns during gait in children with cerebral palsy: A Delphi consensus study. Dev. Med. Child Neurol. 2016, 58, 306–313. [Google Scholar] [CrossRef] [PubMed]
  16. Soska, K.C.; Seisler, A.R.; Gilmore, R.O. Databrary & Datavyu Help Videos 2018. Available online: https://nyu.databrary.org/volume/686 (accessed on 25 November 2024).
  17. Steffan, A.; Zimmer, L.; Arias-Trejo, N.; Bohn, M.; Dal Ben, R.; Flores-Coronado, M.A.; Franchin, L.; Garbisch, I.; Grosse Wiesmann, C.; Hamlin, J.K.; et al. Validation of an open source, remote web-based eye-tracking method (WebGazer) for research in early childhood. Infancy 2024, 29, 31–55. [Google Scholar] [CrossRef] [PubMed]
  18. Dickie, D.A.; Shenkin, S.D.; Anblagan, D.; Lee, J.; Cabez, M.B.; Rodriguez, D.; Boardman, J.P.; Waldman, A.; Job, D.E.; Wardlaw, J.M. Whole brain magnetic resonance image atlases: A systematic review of existing atlases and caveats for use in population imaging. Front. Neuroinform. 2017, 11, 1. [Google Scholar] [CrossRef]
  19. Easterby, R.; Kroemer, K.H.E.; Chaffin, D.B. (Eds.) Anthropometry and Biomechanics; Springer: Boston, MA, USA, 1982; ISBN 978-1-4684-1100-3. [Google Scholar]
  20. Sermpon, N.; Gima, H. Correlation between pose estimation features regarding movements towards the midline in early infancy. PLoS ONE 2024, 19, e0299758. [Google Scholar] [CrossRef] [PubMed]
  21. O’Brien, M.K.; Hohl, K.; Lieber, R.L.; Jayaraman, A. Automate, Illuminate, Predict: A Universal Framework for Integrating Wearable Sensors in Healthcare. Digit. Biomarkers 2024, 8, 149–158. [Google Scholar] [CrossRef]
  22. Deng, W.; Sargent, B.; Havens, K.; Vanderbilt, D.; Rosales, M.; Pulido, J.C.; Matarić, M.J.; Smith, B.A. Correlation between performance and quantity/variability of leg exploration in a contingency learning task during infancy. Infant Behav. Dev. 2023, 70, 101788. [Google Scholar] [CrossRef]
  23. Shida-Tokeshi, J.; Lane, C.J.; Trujillo-Priego, I.A.; Deng, W.; Vanderbilt, D.L.; Loeb, G.E.; Smith, B.A. Relationships between full-day arm movement characteristics and developmental status in infants with typical development as they learn to reach: An observational study. Gates Open Res. 2018, 2, 17. [Google Scholar] [CrossRef]
  24. Smith, B.; Vanderbilt, D.; Applequist, B.; Kyvelidou, A. Sample Entropy Identifies Differences in Spontaneous Leg Movement Behavior between Infants with Typical Development and Infants at Risk of Developmental Delay. Technologies 2017, 5, 55. [Google Scholar] [CrossRef] [PubMed]
  25. Rosales, M.R.; Deng, W.; Nishiyori, R.; Vanderbilt, D.L.; Smith, B.A. Leg Movement Rate before and after a Caregiver-Provided Intervention for Infants at Risk of Developmental Disability: A Pilot Study. Phys. Occup. Ther. Pediatr. 2022, 42, 259–274. [Google Scholar] [CrossRef]
  26. Khasgiwale, R.N.; Smith, B.A.; Looper, J. Leg Movement Rate Pre- and Post-Kicking Intervention in Infants with Down Syndrome. Phys. Occup. Ther. Pediatr. 2021, 41, 590–600. [Google Scholar] [CrossRef]
  27. Chen, C.Y.; Harrison, T.; Heathcock, J. Infants with complex congenital heart diseases show poor short-term memory in the mobile paradigm at 3 months of age. Infant Behav. Dev. 2015, 40, 12–19. [Google Scholar] [CrossRef]
  28. Campbell, S.K.; Infant Motor Performance Scales, LLC. The Test of Infant Motor Performance: Test User’s Manual Version 3.0 for the TIMP Version 5; Infant Motor Performance Scales, LLC: Glenview, IL, USA, 2012. [Google Scholar]
  29. Bayley, N. Bayley Scale of Infant and Toddler Development, Third Edition (Bayley-3), 3rd ed.; NCS Pearson: Bloomington, MN, USA, 2006. [Google Scholar]
  30. Lugaresi, C.; Tang, J.; Nash, H.; McClanahan, C.; Uboweja, E.; Hays, M.; Zhang, F.; Chang, C.-L.; Yong, M.G.; Lee, J.; et al. MediaPipe: A Framework for Building Perception Pipelines. arXiv 2019, arXiv:1906.08172. [Google Scholar]
  31. Knotions. Children’s Body Measurements. Available online: https://knotions.com/resources/standard-measurements/childrens-body-measurements/ (accessed on 1 May 2024).
  32. Fitter, N.T.; Funke, R.; Pulido, J.C.; Eisenman, L.E.; Deng, W.; Rosales, M.R.; Bradley, N.S.; Sargent, B.; Smith, B.A.; Mataric, M.J. Socially Assistive Infant-Robot Interaction: Using Robots to Encourage Infant Leg-Motion Training. IEEE Robot. Autom. Mag. 2019, 26, 12–23. [Google Scholar] [CrossRef]
  33. Richman, J.S.; Moorman, J.R. Physiological time-series analysis using approximate entropy and sample entropy maturity in premature infants Physiological time-series analysis using approximate entropy and sample entropy. Am. J. Physiol. Heart Circ. Physiol. 2000, 278, H2039–H2049. [Google Scholar] [CrossRef]
  34. Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; L. Erlbaum Associates: Hillsdale, NJ, USA, 1988; ISBN 9781134742707. [Google Scholar]
  35. Smith, B.A. Full-day arm movement data across infancy. In Journal of Sport and Exercise Psychology, Proceedings of the North American Society for the Psychology of Sport and Physical Activity, Montreal, QC, Canada, 15–18 June 2016; Human Kinetics: Champaign, IL, USA, 2016; p. s141. [Google Scholar]
  36. Abrishami, M.S.; Nocera, L.; Mert, M.; Trujillo-Priego, I.A.; Purushotham, S.; Shahabi, C.; Smith, B.A. Identification of developmental delay in infants using wearable sensors: Full-day leg movement statistical feature analysis. IEEE J. Transl. Eng. Health Med. 2019, 7, 2800207. [Google Scholar] [CrossRef]
  37. Goodfellow, D.; Zhi, R.; Funke, R.; Pulido, J.C.; Mataric, M.; Smith, B.A. Predicting Infant Motor Development Status using Day Long Movement Data from Wearable Sensors. arXiv 2018, arXiv:1807.02617. [Google Scholar]
  38. Deng, W.; Vanderbilt, D.L.; Smith, B.A. Differences in spontaneous leg movement patterns between infants with typical development and infants at risk for developmental delay: Cross-sectional observation prior to sitting onset. J. Mot. Learn. Dev. 2018, 6, 101–113. [Google Scholar] [CrossRef]
  39. Ulrich, B.D. Opportunities for Early Intervention Based on Theory, Basic Neuroscience, and Clinical Science. Phys. Ther. 2010, 90, 1868–1880. [Google Scholar] [CrossRef]
  40. Lobo, M.A.; Harbourne, R.T.; Dusing, S.C.; McCoy, S.W. Grounding early intervention: Physical therapy cannot just be about motor skills anymore. Phys. Ther. 2013, 93, 94–103. [Google Scholar] [CrossRef]
  41. Trujillo-Priego, I.A.; Smith, B.A. Kinematic characteristics of infant leg movements produced across a full day. J. Rehabil. Assist. Technol. Eng. 2017, 4, 205566831771746. [Google Scholar] [CrossRef] [PubMed]
  42. Harbourne, R.T.; Lobo, M.A.; Karst, G.M.; Galloway, J.C. Sit happens: Does sitting development perturb reaching development, or vice versa? Infant Behav. Dev. 2013, 36, 438–450. [Google Scholar] [CrossRef]
  43. Lobo, M.A.; Galloway, J.C.; Heathcock, J.C. Characterization and intervention for upper extremity exploration and reaching behaviors in infancy. J. Hand Ther. 2015, 28, 114–125. [Google Scholar] [CrossRef] [PubMed]
  44. Oh, J.; Ordoñez, E.L.T.; Velasquez, E.; Mejía, M.; Grazioso, M.P.; Rohloff, P.; Smith, B.A. Early full-day leg movement kinematics and swaddling patterns in infants in rural Guatemala: A pilot study. PLoS ONE 2024, 19, e0298652. [Google Scholar] [CrossRef]
  45. Liao, Y.; Skelton, K.; Dunton, G.; Bruening, M. A systematic review of methods and procedures used in ecological momentary assessments of diet and physical activity research in youth: An adapted STROBE checklist for reporting EMA Studies (CREMAS). J. Med. Internet Res. 2016, 18, e151. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Sample image of all 33 anatomical landmarks on a child. The added number indicates the following virtual markers used for the study: (1) right shoulder, (2) left shoulder, (3) right wrist, (4) left wrist, (5) right ankle, and (6) left ankle.
Figure 1. Sample image of all 33 anatomical landmarks on a child. The added number indicates the following virtual markers used for the study: (1) right shoulder, (2) left shoulder, (3) right wrist, (4) left wrist, (5) right ankle, and (6) left ankle.
Sensors 24 07586 g001
Figure 2. Example of movement count for a single limb. In this case, we display the movement of the left ankle. Triangles represent each occurrence of a movement.
Figure 2. Example of movement count for a single limb. In this case, we display the movement of the left ankle. Triangles represent each occurrence of a movement.
Sensors 24 07586 g002
Figure 3. Box plots for leg (A) and arm (B) movement rate between infants with (CCHD) and without (TD) complex congenital heart disease. Dots on each plot represent average individual movement rate data. Edges of the box represent the 25th and 75th percentile, the middle line in the box is the median, and whiskers represent maximum and minimum values.
Figure 3. Box plots for leg (A) and arm (B) movement rate between infants with (CCHD) and without (TD) complex congenital heart disease. Dots on each plot represent average individual movement rate data. Edges of the box represent the 25th and 75th percentile, the middle line in the box is the median, and whiskers represent maximum and minimum values.
Sensors 24 07586 g003
Figure 4. Box plots of sample entropy for leg and arm movements for infants with (CCHD) and without (TD) complex congenital heart disease. The side and limb for each plot is as follows: (A) left leg, (B) right leg, (C) left arm, (D) right arm. Dots on each plot represent individual data. Edges of the box represent the 25th and 75th percentile, the middle line in the box is the median, and whiskers represent maximum and minimum values. A plus represents a potential outlier according to the MATLAB function.
Figure 4. Box plots of sample entropy for leg and arm movements for infants with (CCHD) and without (TD) complex congenital heart disease. The side and limb for each plot is as follows: (A) left leg, (B) right leg, (C) left arm, (D) right arm. Dots on each plot represent individual data. Edges of the box represent the 25th and 75th percentile, the middle line in the box is the median, and whiskers represent maximum and minimum values. A plus represents a potential outlier according to the MATLAB function.
Sensors 24 07586 g004
Table 2. Two-by-two contingency tables comparing the agreement and disagreements between the detection of leg movements using the pose estimation and behavioral coding for the following: (a) the combination of all children, (b) children with typical development, and (c) children with complex congenital heart disease. The section with no movement (i.e., the cell with “No behavioral coded movement” and “No pose estimation Movement”) is left blank because it is not possible to calculate the number of movements that did not happen.
Table 2. Two-by-two contingency tables comparing the agreement and disagreements between the detection of leg movements using the pose estimation and behavioral coding for the following: (a) the combination of all children, (b) children with typical development, and (c) children with complex congenital heart disease. The section with no movement (i.e., the cell with “No behavioral coded movement” and “No pose estimation Movement”) is left blank because it is not possible to calculate the number of movements that did not happen.
(a)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement25531
No pose estimation movement16
(b)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement16218
No pose estimation movement15
(c)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement9313
No pose estimation movement1
Table 3. Two-by-two contingency tables comparing the agreement and disagreements between the detection of arm movements using the pose estimation and behavioral coding for the following: (a) the combination of all children, (b) children with typical development, and (c) children with complex congenital heart disease. The section with no movement (i.e., the cell with “No behavioral coded movement” and “No pose estimation Movement”) is left blank because it is not possible to calculate the number of movements that did not happen.
Table 3. Two-by-two contingency tables comparing the agreement and disagreements between the detection of arm movements using the pose estimation and behavioral coding for the following: (a) the combination of all children, (b) children with typical development, and (c) children with complex congenital heart disease. The section with no movement (i.e., the cell with “No behavioral coded movement” and “No pose estimation Movement”) is left blank because it is not possible to calculate the number of movements that did not happen.
(a)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement24426
No pose estimation movement14
(b)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement18319
No pose estimation movement12
(c)
Behavioral coded movementNo behavioral coded movement
Pose estimation movement617
No pose estimation movement2
Table 4. Spearman correlation results when comparing average leg and arm movement rate with the Bayley Scale for Infant and Toddler Development’s motor scale and Test for Infant Motor Performance (TIMP).
Table 4. Spearman correlation results when comparing average leg and arm movement rate with the Bayley Scale for Infant and Toddler Development’s motor scale and Test for Infant Motor Performance (TIMP).
Movement VariableDevelopmental Assessment rp
Leg movement rateBayley—Cognitiver = −0.15p = 0.64
Bayley—Communicationr = 0.33p = 0.29
Bayley—Motorr = −0.01p = 0.97
TIMP r = 0.47p = 0.12
Arm movement rateBayley—Cognitiver = −0.34p = 0.28
Bayley—Communicationr = 0.60p = 0.04
Bayley—Motor r = 0.10p = 0.76
TIMP r = 0.23p = 0.35
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Rosales, M.R.; Simsic, J.; Kneeland, T.; Heathcock, J. Quantifying Arm and Leg Movements in 3-Month-Old Infants Using Pose Estimation: Proof of Concept. Sensors 2024, 24, 7586. https://doi.org/10.3390/s24237586

AMA Style

Rosales MR, Simsic J, Kneeland T, Heathcock J. Quantifying Arm and Leg Movements in 3-Month-Old Infants Using Pose Estimation: Proof of Concept. Sensors. 2024; 24(23):7586. https://doi.org/10.3390/s24237586

Chicago/Turabian Style

Rosales, Marcelo R., Janet Simsic, Tondi Kneeland, and Jill Heathcock. 2024. "Quantifying Arm and Leg Movements in 3-Month-Old Infants Using Pose Estimation: Proof of Concept" Sensors 24, no. 23: 7586. https://doi.org/10.3390/s24237586

APA Style

Rosales, M. R., Simsic, J., Kneeland, T., & Heathcock, J. (2024). Quantifying Arm and Leg Movements in 3-Month-Old Infants Using Pose Estimation: Proof of Concept. Sensors, 24(23), 7586. https://doi.org/10.3390/s24237586

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop