[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
TruthTrust: Truth Inference-Based Trust Management Mechanism on a Crowdsourcing Platform
Next Article in Special Issue
Multi-Channel Fusion Classification Method Based on Time-Series Data
Previous Article in Journal
Developing a Modern Greenhouse Scientific Research Facility—A Case Study
Previous Article in Special Issue
A Wearable Navigation Device for Visually Impaired People Based on the Real-Time Semantic Visual SLAM System
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 21
Issue 8
10.3390/s21082576
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Multi-DoF Prosthetic Hand Finger Joint Controller for Wearable sEMG Sensors by Nonlinear Autoregressive Exogenous Model

by
Zhaolong Gao
1,
Rongyu Tang
2,*,
Qiang Huang
2 and
Jiping He
2
1
Key Laboratory of Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China
2
Beijing Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing 100081, China
*
Author to whom correspondence should be addressed.
Sensors 2021, 21(8), 2576; https://doi.org/10.3390/s21082576
Submission received: 7 March 2021 / Revised: 26 March 2021 / Accepted: 31 March 2021 / Published: 7 April 2021
(This article belongs to the Special Issue Wearable Sensor for Activity Analysis and Context Recognition)
Browse Figures
Figure 1
<p>Examples of commercial surface electromyography (sEMG) sensors. (<b>a</b>) Delsys Trigno (<a href="http://www.delsys.com/" target="_blank">http://www.delsys.com/</a>, accessed on 25 October 2020). (<b>b</b>) Cometa Mini Wave (<a href="http://www.cometasystems.com/" target="_blank">http://www.cometasystems.com/</a>, accessed on 25 October 2020). (<b>c</b>) Myo armband (<a href="http://www.thalmic.com/" target="_blank">http://www.thalmic.com/</a>, accessed on 25 January 2018). (<b>d</b>) Ottobock 13E200 (<a href="http://www.ottobock.com/" target="_blank">http://www.ottobock.com/</a>, accessed on 25 October 2020).</p> "> Figure 2
<p>Typical top-level controller of an sEMG-based prosthesis.</p> "> Figure 3
<p>Myo armband and dataglove setup in data acquisition experiments [<a href="#B33-sensors-21-02576" class="html-bibr">33</a>] (Adapted with permission license CC BY 4.0 (2017)).</p> "> Figure 4
<p>Strain gauge placement of the CyberGlove II.</p> "> Figure 5
<p>Block diagram of the nonlinear autoregressive exogenous (NARX) model-based top-level controller.</p> "> Figure 6
<p>Representative EMG signals and dataglove signals (A.U.: Arbitrary Units).</p> "> Figure 7
<p>Diagram of the NARX model network structure.</p> "> Figure 8
<p>Training setup in MATLAB. The number below each block represents the dimensions. <math display="inline"><semantics> <mrow> <mi>x</mi> <mfenced> <mi>t</mi> </mfenced> </mrow> </semantics></math> contains RMS and zero crossing (ZC) with 16 dimensions of each, and the <math display="inline"><semantics> <mrow> <mi>y</mi> <mfenced> <mi>t</mi> </mfenced> </mrow> </semantics></math> is the output feedback of 6. The delay was set to 4. The value of b in the hidden layer and the output layer was set to 0 initially and updated during the training.</p> "> Figure 9
<p>Mean square error with different numbers of neurons. The dashed lines are MSE of different subjects. The average performance was calculated from all subjects.</p> "> Figure 10
<p>Error autocorrelation of the proposed NARX model.</p> "> Figure 11
<p>Comparison of output signals after post-processing.</p> "> Figure 12
<p>Comparison of NARX outputs and target joint angles in reduced space (all channels).</p> "> Figure 13
<p>Comparison of NARX outputs and target joint angles in reduced space (single channel).</p> "> Figure 14
<p>Performance of the proposed model referring to individual movements of fingers.</p> "> Figure 15
<p>Comparison of NARX model output and the target in daily movements.</p> "> Figure 16
<p>Mean square error of daily functional movements.</p> ">
Versions Notes

Abstract

:
The loss of mobility function and sensory information from the arm, hand, and fingertips hampers the activities of daily living (ADL) of patients. A modern bionic prosthetic hand can compensate for the lost functions and realize multiple degree of freedom (DoF) movements. However, the commercially available prosthetic hands usually have limited DoFs due to limited sensors and lack of stable classification algorithms. This study aimed to propose a controller for finger joint angle estimation by surface electromyography (sEMG). The sEMG data used for training were gathered with the Myo armband, which is a commercial EMG sensor. Two features in the time domain were extracted and fed into a nonlinear autoregressive model with exogenous inputs (NARX). The NARX model was trained with pre-selected parameters using the Levenberg–Marquardt algorithm. Comparing with the targets, the regression correlation coefficient (R) of the model outputs was more than 0.982 over all test subjects, and the mean square error was less than 10.02 for a signal range in arbitrary units equal to [0, 255]. The study also demonstrated that the proposed model could be used in daily life movements with good accuracy and generalization abilities.

1. Introduction

In the United States, an estimated 1.6 million people live with limb loss, and 50,000–100,000 new amputations occur every year. Around one fourth of the limb amputees have upper-limb amputation [1]. The amputation results in the loss of mobility function and sensory information of the arm, hand, and fingertips, which severely compromises the activities of daily living (ADL) of the amputees. In addition to the high prevalence of amputation, the demand for improved upper extremity prostheses has increased. However, the traditional body-powered prostheses, which rely on a cable system to move, allow only limited degree of freedom (DoF) movement. With the recent development of robotics and bioinformatics, bionic prostheses have shown promising potential for rehabilitation and improving the ADL qualities of amputees. The bionic prostheses, powered by electronic motors, can perform various motions with multiple modes and DoFs. However, most of them are confined to the laboratory environment; the commercially available bionic prostheses usually have only limited numbers of DoFs. This has compromised the use of bionic prostheses and led to a high abandon rate [2]. The limitation of DoF is mainly caused by poor signal quality and controller algorithms. A real-time multiple DoF control method for a commercially available bionic prosthesis still needs exploration.
The inputs of the controller of bionic prostheses are obtained from biological signals of the user and translate into commands of prostheses with the help of a human–machine interface (HMI). HMI enables the amputees to interact with robots and prostheses to recover or compensate for their lost function. For HMIs in bionic prostheses, the applicable biological signals range from biomechanical parameters of the limb to the electrophysiological activities in the neural system. In the last few years, electroencephalogram (EEG), electromyography (EMG), and various types of biological signals have been applied in HMIs [3,4,5]. Collinger et al. proposed a method to control neuro-prosthesis by decoding the activity of the motor cortex neurons [6]. Participants with tetraplegia successfully controlled a prosthetic limb of seven DoFs via brain–machine interfaces, which related the firing rate of the neurons to the movement velocity of a modular prosthetic limb. Smith et al. achieved simultaneous proportional control of a prosthetic wrist and hand with multiple DoFs [7]. The input signals were gathered by intramuscular EMG with dual-site configuration. However, these approaches are invasive and usually involve surgical procedures for electrode implantation. The surgery of implantation limits the usability of invasive HMIs [8,9]. The risk of infectiousness during the process, contingent rejection reaction, potential deterioration of signal qualities, and other medical concerns challenge the acceptance of invasive HMIs. These disadvantages compromise the application of invasive HMIs in daily-use prostheses requiring a presumably long service life. With these considerations, the non-invasive approaches are superior, especially in terms of commercially available extremity prostheses.
The non-invasive electrophysiological-based HMI uses neural activity signals, such as EEG, EMG, etc., as input signals [10,11]. In particular, surface electromyography (sEMG) is a choice applied with a long history. Surface EMG signals are formed by the variations in states of skeletal muscles, which are triggered by the human intention transferred from neurons to corresponding limbs. The human intention can be reflected directly with no loss and low latency. Therefore, sEMG signals are chosen to be an effective source for prostheses control due to the noninvasiveness and rich information [12]. Compared with other non-invasive approaches such as sonomyography (SMG) [13] and steady-state visual evoked potential [14], the size and weight of the sensors for sEMG acquisition are more suitable for wearable requirements [15] and compact enough for fitting inside a prosthetic hand device [16], as shown in Figure 1. However, this also limits the quality of the acquired signal due to the geometric requirements. To overcome this dilemma, a control system that takes the signal quality of the sEMG sensor under geometric constraints into account needs to be investigated.
For a general control architecture of sEMG-based prostheses, mainly three levels of components are used [17]: top-level, mid-level, and low-level controllers. The top-level controller concentrates on interpreting human intention. The mid-level controller translates the human intention into different motion modes. The low-level controller determines the input parameters of the motor of the prosthesis. The three-level architecture links the raw electrophysiological signals to the control signals of the actuator.
In commercially available prostheses, the top-level controller consists of several components, as shown in Figure 2. The signal processing module denoises the input signal and is usually accompanied by dimensional reduction. The feature extraction module calculates the hand-crafted features for classification, and finally the classifier produces the results from the windowed features for the mid-level controller. Two categories of controllers can be distinguished based on the outputs of the classification module. The first approach uses a processed sEMG signal as a trigger for different movement modes. In this approach, researchers use sEMG signals to recognize limb motion and try to achieve more modes and higher recognition rates. Hence, studies concentrated on feature extraction methods and classification algorithms [18,19,20,21]. However, the mode switching methods are limited by the number of modes available. The predetermined modes of movements cannot reflect the user’s intended motion precisely. As a result, the embodiment of the control and the smoothness of motion are compromised. The second approach is to continuously estimate the motion variables from human intention to achieve smooth control of the prosthetic hand. For instance, many methods attempted to relate the sEMG signals to the joint angles of the prosthetic limb. Lloyd and Besier constructed a forward biomechanical model for joint torque calculation with sEMG signals [22]. The mapping of sEMG signals to joint variables such as torque and angles can also be achieved with polynomial fitting [23] or neural network [24,25,26]. In these methods, the motion control of the prosthesis is continuous allowing more dexterous movements and the trajectory is not confined to pre-set modes. These continuous methods have some limitations. For example, the body parameters are highly user dependent and hence limiting the generalization capability and requiring a large data set for training.
In commercial applications, a variety of sophisticated prostheses choose sEMG as their input signal. While significant improvement has been achieved over the past decades, the controllable DoFs are still limited. For instance, the Otto Bock’s Sensorhand Speed [27] only enforced single-DoF grippers. It was equipped with two sEMG electrodes [28] placed on the loci of the residual muscle with maximum electrophysiological activity. The limited DoFs compromised the use of sEMG-based prosthesis. Meanwhile, the unreliable signals of the wearable sEMG sensors were due to the intrinsic variety of neural activities, electrode shift, sweat, and muscle fatigue [29]. As a result, the abandonment rate of hand prostheses remains high [30,31]. Additionally, the features affected the accuracy of the classification controller significantly. As a result, the feature selection is of vital importance for sEMG-based HMI. However, the hand-crafted features compromise the generalization ability of the HMI. To avoid feature selection, raw sEMG signals should be fed directly into the estimation algorithm, and hence the feature selection and processing can be done inside the controller. Thus, accurate and fast feature selection and extraction for joint angle estimation are still considered a challenging task for commercially available extremity prosthesis.
With the recent advent of multi-DoF hand prostheses, the need for a dexterous multiple DoFs top-level controller for HMI has become urgent. In an ideal controller, every degree of freedom is proportionally controllable by the user intent independently. Many studies have been conducted to augment the performance of sEMG signals in terms of acquisition, denoising, and classification. New advances in classification algorithms have been made with the help of machine learning. Furui et al. [32] employed an sEMG-based motion generator to realize the control of multiple finger movements, with the introduction of advanced control mechanisms based on human motion. Both healthy and upper-limb amputation individuals participated in the evaluation test.
In this study, a nonlinear autoregressive exogenous (NARX)-based model was proposed to estimate finger joint angles which could be used as targets for the prosthetic hand top-level controller. The training and test data used in the experiment were from the 5th NinaPro database which focused on the signals of wearable sEMG sensors suitable for a commercially available prosthetic hand. In this database, a Myo armband was used as the sEMG acquisition device. The Myo armband could work without extra batteries and transfer data wirelessly with adequate sample frequency and accuracy. This database established a platform for sEMG-based controller design. First, the sEMG signals were fed into the preprocessing module, by which the sEMG signals were segmented and their dimension was reduced by applying Principal Component Analysis (PCA). The features were extracted in the time domain. Then, an NARX network was employed to select and map the optimal sEMG features to the pre-chosen space of joint angles. Finally, an inverse PCA and Kalman filter were applied for post-processing to convert the reduced space into the original joint angle space after signal conditioning. The model proposed in this study outperforms the multilayer perceptron neural networks. It demonstrated a good generalization ability to simplify the learning process for users. It aimed to realize the continuous sEMG control in wearable devices for HMIs and enhance the stability in extremity prosthesis.
This manuscript is organized as follow. Section 2.1 presents the details of data acquisition and the preprocessing of the NinaPro database of finger movements used in this study. The background and implementation details of the NARX controller are presented in Section 2.2. The results and discussion are presented in Section 3 followed by the conclusion of this study in Section 4.

2. Materials and Methods

2.1. NinaPro Database

In this study, the NinaPro database was used for training and validation of the algorithm. The database contained data from different acquisition setups. Among these, the Double Myo dataset (DB5), which used Myo armband (Thalmic Labs, Kitchener, ON, Canada) as the acquisition device, was chosen to simulate the input signal from wearable sEMG sensors of commercial prosthetic hands. The electrode placement and the signal quality of DB5 were similar to those applied in the commercially available prostheses, making the database particularly suitable for the exploration of sEMG recognition in commercially available devices.

2.1.1. Subjects and Acquisition Setups

The experiment data of 10 intact participants (8 males and 2 females) were included in DB5. The sEMG signals were recorded with two Myo armbands around the forearm of the participants as shown in Figure 3. There are eight stainless-steel single differential electrodes, which could be evenly placed around the forearm of the user. The placement of the upper armband was closer to the elbow, with the first electrode on the radio humeral joint and the lower one next to the first with 22.5 degrees rotation. The sEMG was sampled by the electrodes of the Myo armband at 200 Hz. The data stream was transferred to the computer via Bluetooth. The configuration of the Double Myo armband was a uniform and affordable platform for muscle exploration and mapping and algorithms of the top-level controller in a commercial prosthetic hand.
The kinematic data of the hand were recorded using a CyberGlove II dataglove (CyberGlove Systems LLC, San Jose, CA, USA) with 22 strain gauge sensors. The CyberGlove captured the motion of fingers by measuring the joint angles, as shown in Figure 4. The return value of the strain gauge was proportional to the joint angle with an 8-bit resolution. The data of sEMG signals from Myo armband and the kinematic data from CyberGlove II are all dimensionless. Their range is [0, 255].

2.1.2. Acquisition Protocol

The participants in the DB5 database were requested to perform three kinds of exercises: 52 movements including basic finger movement (Exercise 1), isometric/isotonic hand configurations (Exercise 2), and functional movements in daily life (Exercise 3) [34]. They were asked to follow the movements represented by a screen in front of them.
The movements selected for model training were the signals of Exercise 1 in the protocol. The movements in this exercise were basic movements of flexion and extension of each finger. The details of the movements are described in [34].

2.2. NARX Controller

The block diagram of the proposed top-level controller is shown in Figure 5. The raw sEMG data collected from the sensors of the prosthetic hand were sent to the feature extraction module after preprocessing. Then, the features were extracted to construct the input vector for the NARX model. The number of output dimensions of the NARX model was consistent with the DoFs of the prosthetic hand after dimension reduction by PCA. The output of the NARX model was reconstructed into the original dimensions of the controllable joints of the prosthetic hand via inverse PCA. Post-processing was performed to ensure the continuity and robustness of the output target angles in the case of dangerous movements.

2.2.1. Data Preprocessing

The raw sEMG data from the Myo armband and joint angle data from Cyberglove II needed to be processed before NARX model training. The sEMG signal and the joint angle data were merged and synchronized according to the timestamp and resampled to 200 Hz.
  • Surface EMG Signal: The powerline interference was removed by a 50 Hz notch filter. In NinaPro DB5, the labels of the movements were also recorded along with the sEMG signals. The labels were corrected with algorithms described in [35,36] for synchronization and removal of errors.
  • Dataglove Signal: For the CyberGlove joint angle data, PCA [37] was applied to the data for dimension reduction. The first six components with the largest variances, that is, the eigenvalues of the covariance matrix, were chosen as the target controller outputs. The PCA algorithm converted the original dataset into a new space allowing the control of multiple DoFs with limited controller outputs. In this study, the first six PCs were considered significant to the multiple DoF spaces. By employing the “inverse PCA” algorithm, the controller could remap PCs to the original DoFs of the prosthetic hand using the PC matrix obtained from experimental data before generating the final outputs. Thus, all DoFs of a dexterous prosthesis might be controlled in synergy with the constraints of sEMG signals. This method was implemented in the reconstruction module, as shown in Figure 5. The signals after preprocessing are shown in Figure 6.

2.2.2. Feature Extraction

Before feeding into the NARX network, the features needed to be computed from the preprocessed sEMG signals. However, the extraction of user intention from the raw sEMG signal was of high computational complexity due to the intrinsic attributes of sEMG signals, which were stochastic, complex, and nonlinear. This limited the use of sEMG signals in real-time oriented applications [38].
Data segmentation was conducted prior to feature extraction. The time windows were set to 200 ms to avoid high computational costs. The segmentation in this controller was based on non-overlapping sliding windows to enhance the real-time processing ability of the controller. The interval of the sliding window was set to 200 ms to meet the requirement of real-time control with acceptable latency.
The features employed in this application were time-domain statistical values. A lot of them could be used as features for the classification of hand movement. The features chosen in this study were root mean square (RMS) and zero crossing (ZC). The equation for RMS and ZC are given as (1) and (2), separately, and the details are described in [39].
RMS = 1 n ∑ k = 1 N x k 2
ZC = ∑ k = 1 N s i g n x k x k + 1   a n d   x k x k + 1 ≥ t h r e s h o l d   v a l u e
Due to the limitation of the computational complexity, the feature selected for the real-time controller should cover as many aspects of the data characteristics as possible within adequate dimensions. The two features implemented in this study were both in the time domain to obtain a low reaction time and they covered different aspects of characteristics of the data: RMS related to the amplitude in the time domain and the ZC related to the oscillation frequency.

2.2.3. NARX Model

Neural networks are effective for the pattern classification of unstructured static data, which are time-invariant. In temporal pattern recognition, patterns change over time. Traditional feedforward neural networks, however, are not effective for temporally evolving data. Therefore, the alteration of the structure of the neural network should be done to consider temporal dependences during training.
A NARX model was employed in this study for estimating the angular displacement of the fingers. NARX structure-based modeling was developed to capture the nonlinear dynamics of the temporal system [40]. The definition of the model is given by
y t = f ω t
and
ω t = u t u t − 1 ⋮ u t − n u y t − 1 y t − 2 ⋮ y t − n y
where y ( t ) is the estimated output with the input u ( t ) by the regression vector ω ( t ) at time t . u and y are the input and output vectors of the system, respectively [39]. Figure 7 shows the diagram of the NARX model network structure. NARX model network contained a multilayer feedforward network and a recurrent neural network with time delays [41]. The recurrent neural network, in addition to the feedforward neural network, used the output of the last state as the input of the next state in a feedback loop. The proposed model had three layers: (1) Input layer with delays in which the input layer took the inputs u t − n u and the outputs y t − n y from the past, whose numbers were determined beforehand. (2) Hidden layer for feature selection and processing in which the input features were RMS and ZC, and the outputs were the angular displacements of the finger joints. The number of nodes of the hidden layer was chosen by a parameter sweep experiment, which tested the performance of networks with different numbers of hidden nodes. (3) Output layer, which produced the six components in the reduced joint angle space according to the activation of hidden layer nodes. The outputs were fed back to the input layer in the next state.

2.2.4. Training Procedure

The function f in (3) could be approximated by various methods, such as fuzzy methods or neural networks. In the NARX model, the function f was trained with pre-selected parameters using the Levenberg–Marquardt (LM) algorithm. The RMS and ZC features extracted from the sEMG signals were randomly divided into three parts: training (70% of all data), validation (15% of all data), and testing (15% of all data).
The LM algorithm, modified and improved from Newton’s method, combined the advantages of the convergence speed of the Gauss–Newton method and the stability of the steepest descent algorithm. The weight update equation based on the LM algorithm is given by
ω n + 1 − ω n = − J n T x J n x + μ I − 1 J n T x e n x
where J is the Jacobian matrix, e ( x ) is the training error, and I is the identity matrix. The LM algorithm switched between the Gauss–newton method and steepest descent method based on the value of μ . The steepest decent algorithm was activated when μ was large and the Gauss–Newton method was activated when μ was low.
The training NARX model was established using the neural network toolbox in MATLAB (MathWorks, Inc., Natick, CA, USA). The training architecture is shown in Figure 8. The number below each block represents the data dimensions. x t contains RMS and ZC with 16 dimensions of each. They are calculated channel-wise from the 16 channel sEMG signal (8 electrodes times 2 armband). The y t is the output feedback of 6 output values. The delay n u was set to 4. The activation function of the hidden layer was sigmoid function which was defined by
sigmoid x = 1 1 + e − x
The selection of the network parameters and the tuning process were two crucial aspects of achieving optimal performance. The choice of network parameters included the number of neurons and delays. Increasing the number of neurons and delays usually led to an improvement in performance. However, it might also cause the network to over-fit, resulting in low mean square error (MSE) on training data but high MSE on testing data. The network would be considered converged if the performance gradient falls below the pre-selected threshold. Then 6 validation epochs would be performed to ensure the results performance meet the requirement.

2.2.5. Network Parameter Selection

The performance of the neural network model depended on the selection of the input vector and the network parameters. NARX model chose both endogenous and exogenous variables as the input vector, resulting in better performance than using either of them [41]. Since the NARX model was a recurrent neural network, increasing the number of delays in the endogenous inputs and the neurons in the hidden layer usually provided better autocorrelation. However, using a large number of neurons in the network might lead to over-fitting, reducing the performance in out-of-sample data.
In the proposed model, the number of hidden layer neurons was swept from 6 to 20 for model evaluation. The results are shown in Figure 9. It proved that increasing the number of neurons in the hidden layer did not necessarily improve the estimation accuracy of a recurrent network. In the parameter sweep test, the best performance was achieved with around 12 neurons. The MSE did not decrease after increasing the number of neurons. Taking the MSE variance over all subjects into consideration, the choice of 14 neurons reached the minimum value, thus increasing the generalization abilities of the model.
The error autocorrelation was computed to validate the network after determining the number of hidden neurons. The result of the normalized error autocorrelation function is shown in Figure 10. It was observed that the estimation errors were time related. If the estimation model was perfect, it would be only one nonzero value of autocorrelation occurring at zero lag, indicating that the estimated errors had no correlation. On the contrary, if the estimation errors were significant, the results would imply that improvement of the estimation could probably be obtained by increasing delays or modifying the network structure. In the proposed model, the correlations around the zero lag fell in the 95 percent confidence limits, indicating adequate number of delays in the network. As a comprehensive consideration of various factors, the delay of the proposed NARX model was chosen to be 4 with the corresponding number of neurons set to 14.

2.2.6. Post-Processing

The post-processing module had two steps. First was an inverse PCA converter to reconstruct the signal of the joint angle. The second part was the signal conditioning to improve the accuracy of the output by a simple Extended Kalman Filter (EKF) for the noise canceling. Kalman filters have been applied in time varying systems for better performance [42]. The representative results are shown in Figure 11. The overshoot was reduced by the Kalman filter and the reaction time was not undermined.

3. Results and Discussion

3.1. Joint Angle Estimation

The example result of the output of the NARX model compared with the target reduced joint angles is shown in Figure 12 (all 6 output channels) and Figure 13 (single channel) from a subject. The output signals in the rest phase were steady, and the reaction time was kept low when the finger movement was triggered. The output signal had good consistency compared with the original target signal. The MSE and regression value of the model were computed to further evaluate the accuracy of the outputs; the results are shown in Table 1. The results indicated that the model was adequate for a multiple DoF application with accuracy and real-time requirements.
A comparative study of the proposed NARX model with the traditional multilayer perception neural networks (MLPNN) was conducted. The comparison of the performance of the NARX model with that of MLPNN is shown in Table 2 and Table 3. The comparison was performed over all 10 subjects in NinaPro DB5. The MLPNN, which lacked recurrent layers and delay inputs, was outperformed by NARX in both regression and MSE. The detailed performance referring to individual movements of fingers is shown in Figure 14. All the data of NinaPro DB5 were used for performance evaluation. It is worth noting that the model performed better in extension than flexion implying the differences in the amount of information embedded in the muscle activities.

3.2. Generalization Ability

The gathering of training datasets is of vital importance to improve the performance of a neural network model. However, if the algorithm lacks generalization ability, the minimum number of data types for training may be large to cover all possible situations. In this study, the generalization ability of the proposed model was tested by applying the method to hand functional movements in daily life.
In the NinaPro dataset, besides basic finger movements, each participant was required to perform 23 kinds of functional movements in daily activities. In the model training processing, these data were excluded from the training sets on purpose. These datasets were used in this section as the input for the trained NARX network. The comparison of the NARX model output and the target output is shown in Figure 15. The error plot indicates that the maximum error of daily life functional movements samples is less than 13% compared with the target joint angles. The MSE error differed from movements to movements, as shown in Figure 16. It was noteworthy that, approximately, the more the fingers were involved in a single movement the higher the MSE would be. This phenomenon implied that crosstalk might occur between different muscles. The result demonstrated that the pre-training with user data can be done with basic movements solely. Therefore, the user could perform predetermined basic finger movements as the training data for the NARX model controller, which reduced the training process for adoption. The gathering procedure was thus simplified and could be standardized, which was beneficial for reducing the ambiguity in training data. Moreover, the performance could be increased with training data of movements of two fingers simultaneously taking the crosstalk into consideration. As a result, the required time for users to adapt to prosthetic hand movement could be reduced.

4. Conclusions

In this study, an NARX-based HMI top-level controller for finger joint angle estimation from sEMG signals was proposed. The sEMG signals used for training were acquired using Myo armband, and the finger movement data were gathered using dataglove. The obtained sEMG and dataglove signals were synchronized and filtered. Two time-domain features (RMS and ZC) of the sEMG signal were extracted and the target data of finger joints were mapped to a reduced dimension using PCA. The time-domain features were then fed to a NARX model trained using the Levenberg–Marquardt algorithm. The result was evaluated with a correlation coefficient of 0.982 and a mean square error of less than 10.02. The comparison of the performance between the NARX model and ordinary MLPNN model was performed. The results indicated that the model was of better accuracy and adequate for a multiple DoF application. The generalization ability of the model was verified with data from daily functional movements, indicating the model can be trained with data from basic movements solely.

Author Contributions

Conceptualization, Z.G. and R.T.; methodology, writing—original draft preparation Z.G.; writing—review and editing, supervision, R.T., Q.H. and J.H. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by National Key R&D Program of China (Grant No. 2018YFB1307301, 2017YFE0117000), Beijing Municipal Science and Technology Program (Grant No. Z201100004420016).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Ziegler-Graham, K.; MacKenzie, E.J.; Ephraim, P.L.; Travison, T.G.; Brookmeyer, R. Estimating the Prevalence of Limb Loss in the United States: 2005 to 2050. Arch. Phys. Med. Rehabil. 2008, 89, 422–429. [Google Scholar] [CrossRef] [PubMed]
  2. Biddiss, E.A.; Chau, T.T. Upper limb prosthesis use and abandonment: A survey of the last 25 years. Prosthet. Orthot. Int. 2007, 31, 236–257. [Google Scholar] [CrossRef] [PubMed]
  3. Matsuoka, Y.; Afshar, P.; Oh, M. On the design of robotic hands for brain-machine interface. Neurosurg. Focus 2006, 20, 1–9. [Google Scholar] [CrossRef] [PubMed]
  4. Kawamoto, H.; Sankai, Y. Power Assist System HAL-3 for Gait Disorder Person BT—Computers Helping People with Special Needs; Miesenberger, K., Klaus, J., Zagler, W., Eds.; Springer: Berlin/Heidelberg, Germany, 2002; pp. 196–203. [Google Scholar]
  5. Benko, H.; Saponas, T.S.; Morris, D.; Tan, D. Enhancing Input on and above the Interactive Surface with Muscle Sensing. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, Banff, AB, Canada, 23–25 November 2009; pp. 93–100. [Google Scholar]
  6. Collinger, J.L.; Wodlinger, B.; Downey, J.E.; Wang, W.; Tyler-Kabara, E.C.; Weber, D.J.; McMorland, A.J.C.; Velliste, M.; Boninger, M.L.; Schwartz, A.B. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 2013, 381, 557–564. [Google Scholar] [CrossRef] [Green Version]
  7. Smith, L.H.; Kuiken, T.A.; Hargrove, L.J. Real-time simultaneous and proportional myoelectric control using intramuscular EMG. J. Neural. Eng. 2014, 11, 66013. [Google Scholar] [CrossRef] [Green Version]
  8. Waldert, S. Invasive vs. Non-Invasive Neuronal Signals for Brain-Machine Interfaces: Will One Prevail? Front. Neurosci. 2016, 10, 295. [Google Scholar] [CrossRef] [Green Version]
  9. Tahir, A.M. Human machine interface: Robotizing the instinctive living. Int. Robot. Autom. J. 2018, 4, 308–314. [Google Scholar] [CrossRef] [Green Version]
  10. Gomez-Gil, J.; San-Jose-Gonzalez, I.; Nicolas-Alonso, L.F.; Alonso-Garcia, S. Steering a tractor by means of an EMG-based human-machine interface. Sensors 2011, 11, 7110–7126. [Google Scholar] [CrossRef] [Green Version]
  11. Hortal, E.; Planelles, D.; Costa, A.; Iáñez, E.; Úbeda, A.; Azorín, J.M.; Fernández, E. SVM-based Brain–Machine Interface for controlling a robot arm through four mental tasks. Neurocomputing 2015, 151, 116–121. [Google Scholar] [CrossRef]
  12. Thomas, T.M.; Candrea, D.N.; Fifer, M.S.; McMullen, D.P.; Anderson, W.S.; Thakor, N.V.; Crone, N.E. Decoding Native Cortical Representations for Flexion and Extension at Upper Limb Joints Using Electrocorticography. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 27, 293–303. [Google Scholar] [CrossRef]
  13. Zheng, Y.P.; Chan, M.M.F.; Shi, J.; Chen, X.; Huang, Q.H. Sonomyography: Monitoring morphological changes of forearm muscles in actions with the feasibility for the control of powered prosthesis. Med. Eng. Phys. 2006, 28, 405–415. [Google Scholar] [CrossRef] [Green Version]
  14. Allison, B.Z.; Brunner, C.; Kaiser, V.; Müller-Putz, G.R.; Neuper, C.; Pfurtscheller, G. Toward a hybrid brain–computer interface based on imagined movement and visual attention. J. Neural Eng. 2010, 7, 26007. [Google Scholar] [CrossRef]
  15. Ke, A.; Huang, J.; Chen, L.; Gao, Z.; He, J. An Ultra-Sensitive Modular Hybrid EMG–FMG Sensor with Floating Electrodes. Sensors 2020, 20, 4775. [Google Scholar] [CrossRef]
  16. Milosevic, B.; Benatti, S.; Farella, E. Design challenges for wearable EMG applications. In Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE), Lausanne, Switzerland, 27–31 March 2017; pp. 1432–1437. [Google Scholar]
  17. Tucker, M.R.; Olivier, J.; Pagel, A.; Bleuler, H.; Bouri, M.; Lambercy, O.; del R Millán, J.; Riener, R.; Vallery, H.; Gassert, R. Control strategies for active lower extremity prosthetics and orthotics: A review. J. Neuroeng. Rehabil. 2015, 12, 1. [Google Scholar] [CrossRef] [Green Version]
  18. Mesa, I.; Rubio, A.; Tubia, I.; De No, J.; Diaz, J. Channel and feature selection for a surface electromyographic pattern recognition task. Expert Syst. Appl. 2014, 41, 5190–5200. [Google Scholar] [CrossRef]
  19. McCool, P.; Petropoulakis, L.; Soraghan, J.J.; Chatlani, N. Improved pattern recognition classification accuracy for surface myoelectric signals using spectral enhancement. Biomed. Signal Process. Control 2015, 18, 61–68. [Google Scholar] [CrossRef] [Green Version]
  20. Tsai, A.-C.; Luh, J.-J.; Lin, T.-T. A novel STFT-ranking feature of multi-channel EMG for motion pattern recognition. Expert Syst. Appl. 2015, 42, 3327–3341. [Google Scholar] [CrossRef]
  21. Phinyomark, A.; Quaine, F.; Charbonnier, S.; Serviere, C.; Tarpin-Bernard, F.; Laurillau, Y. Feature extraction of the first difference of EMG time series for EMG pattern recognition. Comput. Methods Programs Biomed. 2014, 117, 247–256. [Google Scholar] [CrossRef]
  22. Lloyd, D.G.; Besier, T.F. An EMG-driven musculoskeletal model to estimate muscle forces and knee joint moments in vivo. J. Biomech. 2003, 36, 765–776. [Google Scholar] [CrossRef]
  23. Hayashi, T.; Kawamoto, H.; Sankai, Y. Control method of robot suit HAL working as operator’s muscle using biological and dynamical information. In Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, AB, Canada, 2–6 August 2005; pp. 3063–3068. [Google Scholar]
  24. Zhang, F.; Li, P.; Hou, Z.-G.; Lu, Z.; Chen, Y.; Li, Q.; Tan, M. sEMG-based continuous estimation of joint angles of human legs by using BP neural network. Neurocomputing 2012, 78, 139–148. [Google Scholar] [CrossRef]
  25. Ardestani, M.M.; Zhang, X.; Wang, L.; Lian, Q.; Liu, Y.; He, J.; Li, D.; Jin, Z. Human lower extremity joint moment prediction: A wavelet neural network approach. Expert Syst. Appl. 2014, 41, 4422–4433. [Google Scholar] [CrossRef]
  26. Aung, Y.M.; Al-Jumaily, A. sEMG Based ANN for Shoulder Angle Prediction. Procedia Eng. 2012, 41, 1009–1015. [Google Scholar] [CrossRef] [Green Version]
  27. Amsuess, S.; Goebel, P.; Graimann, B.; Farina, D. Extending mode switching to multiple degrees of freedom in hand prosthesis control is not efficient. In Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA, 26–30 August 2014; pp. 658–661. [Google Scholar]
  28. Merletti, R.; Botter, A.; Troiano, A.; Merlo, E.; Minetto, M.A. Technology and instrumentation for detection and conditioning of the surface electromyographic signal: State of the art. Clin. Biomech. 2009, 24, 122–134. [Google Scholar] [CrossRef] [PubMed]
  29. Ravindra, V.; Castellini, C. A comparative analysis of three non-invasive human-machine interfaces for the disabled. Front. Neurorobot. 2014, 8, 24. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  30. Biddiss, E.; Chau, T. Upper-limb prosthetics: Critical factors in device abandonment. Am. J. Phys. Med. Rehabil. 2007, 86, 977–987. [Google Scholar] [CrossRef] [Green Version]
  31. Peerdeman, B.; Boere, D.; Witteveen, H.; Hermens, H.; Stramigioli, S.; Rietman, H.; Veltink, P.; Misra, S. Myoelectric forearm prostheses: State of the art from a user-centered perspective. J. Rehabil. Res. Dev. 2011, 48, 719–737. [Google Scholar] [CrossRef] [Green Version]
  32. Furui, A.; Eto, S.; Nakagaki, K.; Shimada, K.; Nakamura, G.; Masuda, A.; Chin, T.; Tsuji, T. A myoelectric prosthetic hand with muscle synergy–based motion determination and impedance model–based biomimetic control. Sci. Robot. 2019, 4. [Google Scholar] [CrossRef] [Green Version]
  33. Pizzolato, S.; Tagliapietra, L.; Cognolato, M.; Reggiani, M.; Müller, H.; Atzori, M. Comparison of six electromyography acquisition setups on hand movement classification tasks. PLoS ONE 2017, 12, 1–17. [Google Scholar] [CrossRef] [Green Version]
  34. Atzori, M.; Gijsberts, A.; Heynen, S.; Hager, A.G.M.; Deriaz, O.; Van Der Smagt, P.; Castellini, C.; Caputo, B.; Muller, H. Building the Ninapro database: A resource for the biorobotics community. In Proceedings of the 2012 4th IEEE RAS \& EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, Italy, 24–27 June 2012; pp. 1258–1265. [Google Scholar]
  35. Kuzborskij, I.; Gijsberts, A.; Caputo, B. On the challenge of classifying 52 hand movements from surface electromyography. In Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA, 28 August–1 September 2012; pp. 4931–4937. [Google Scholar]
  36. Staude, G.; Flachenecker, C.; Daumer, M.; Wolf, W. Onset Detection in Surface Electromyographic Signals: A Systematic Comparison of Methods. EURASIP J. Adv. Signal Process. 2001, 2001, 867853. [Google Scholar] [CrossRef] [Green Version]
  37. Pearson, K. On lines and planes of closest fit to systems of points in space. Lond. Edinburgh Dublin Philos. Mag. J. Sci. 1901, 2, 559–572. [Google Scholar] [CrossRef] [Green Version]
  38. Khushaba, R.N.; Kodagoda, S.; Takruri, M.; Dissanayake, G. Toward improved control of prosthetic fingers using surface electromyogram (EMG) signals. Expert Syst. Appl. 2012, 39, 10731–10738. [Google Scholar] [CrossRef]
  39. Hudgins, B.; Parker, P.; Scott, R.N. A new strategy for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 1993, 40, 82–94. [Google Scholar] [CrossRef]
  40. Liu, Y.; Huang, H. Towards a high-stability EMG recognition system for prosthesis control: A one-class classification based non-target EMG pattern filtering scheme. In Proceedings of the 2009 IEEE International Conference on Systems, Man and Cybernetics, San Antonio, TX, USA, 11–14 October 2009; pp. 4752–4757. [Google Scholar]
  41. Mitrea, C.; Lee, C.; Wu, Z. A Comparison between Neural Networks and Traditional Forecasting Methods: A Case Study. Int. J. Eng. Bus. Manag. 2009, 1. [Google Scholar] [CrossRef]
  42. Schimmack, M.; Mercorelli, P.; Maiwald, M. Combining Kalman filter and RLS-algorithm to improve a textile based sensor system in the presence of linear time-varying parameters. In Proceedings of the 2015 17th International Conference on E-health Networking, Application & Services (HealthCom), Boston, MA, USA, 14–17 October 2015; pp. 507–510. [Google Scholar]
Sensors 21 02576 g001 550
Figure 1. Examples of commercial surface electromyography (sEMG) sensors. (a) Delsys Trigno (http://www.delsys.com/, accessed on 25 October 2020). (b) Cometa Mini Wave (http://www.cometasystems.com/, accessed on 25 October 2020). (c) Myo armband (http://www.thalmic.com/, accessed on 25 January 2018). (d) Ottobock 13E200 (http://www.ottobock.com/, accessed on 25 October 2020).
Figure 1. Examples of commercial surface electromyography (sEMG) sensors. (a) Delsys Trigno (http://www.delsys.com/, accessed on 25 October 2020). (b) Cometa Mini Wave (http://www.cometasystems.com/, accessed on 25 October 2020). (c) Myo armband (http://www.thalmic.com/, accessed on 25 January 2018). (d) Ottobock 13E200 (http://www.ottobock.com/, accessed on 25 October 2020).
Sensors 21 02576 g001
Sensors 21 02576 g002 550
Figure 2. Typical top-level controller of an sEMG-based prosthesis.
Figure 2. Typical top-level controller of an sEMG-based prosthesis.
Sensors 21 02576 g002
Sensors 21 02576 g003 550
Figure 3. Myo armband and dataglove setup in data acquisition experiments [33] (Adapted with permission license CC BY 4.0 (2017)).
Figure 3. Myo armband and dataglove setup in data acquisition experiments [33] (Adapted with permission license CC BY 4.0 (2017)).
Sensors 21 02576 g003
Sensors 21 02576 g004 550
Figure 4. Strain gauge placement of the CyberGlove II.
Figure 4. Strain gauge placement of the CyberGlove II.
Sensors 21 02576 g004
Sensors 21 02576 g005 550
Figure 5. Block diagram of the nonlinear autoregressive exogenous (NARX) model-based top-level controller.
Figure 5. Block diagram of the nonlinear autoregressive exogenous (NARX) model-based top-level controller.
Sensors 21 02576 g005
Sensors 21 02576 g006 550
Figure 6. Representative EMG signals and dataglove signals (A.U.: Arbitrary Units).
Figure 6. Representative EMG signals and dataglove signals (A.U.: Arbitrary Units).
Sensors 21 02576 g006
Sensors 21 02576 g007 550
Figure 7. Diagram of the NARX model network structure.
Figure 7. Diagram of the NARX model network structure.
Sensors 21 02576 g007
Sensors 21 02576 g008 550
Figure 8. Training setup in MATLAB. The number below each block represents the dimensions. x t contains RMS and zero crossing (ZC) with 16 dimensions of each, and the y t is the output feedback of 6. The delay was set to 4. The value of b in the hidden layer and the output layer was set to 0 initially and updated during the training.
Figure 8. Training setup in MATLAB. The number below each block represents the dimensions. x t contains RMS and zero crossing (ZC) with 16 dimensions of each, and the y t is the output feedback of 6. The delay was set to 4. The value of b in the hidden layer and the output layer was set to 0 initially and updated during the training.
Sensors 21 02576 g008
Sensors 21 02576 g009 550
Figure 9. Mean square error with different numbers of neurons. The dashed lines are MSE of different subjects. The average performance was calculated from all subjects.
Figure 9. Mean square error with different numbers of neurons. The dashed lines are MSE of different subjects. The average performance was calculated from all subjects.
Sensors 21 02576 g009
Sensors 21 02576 g010 550
Figure 10. Error autocorrelation of the proposed NARX model.
Figure 10. Error autocorrelation of the proposed NARX model.
Sensors 21 02576 g010
Sensors 21 02576 g011 550
Figure 11. Comparison of output signals after post-processing.
Figure 11. Comparison of output signals after post-processing.
Sensors 21 02576 g011
Sensors 21 02576 g012 550
Figure 12. Comparison of NARX outputs and target joint angles in reduced space (all channels).
Figure 12. Comparison of NARX outputs and target joint angles in reduced space (all channels).
Sensors 21 02576 g012
Sensors 21 02576 g013 550
Figure 13. Comparison of NARX outputs and target joint angles in reduced space (single channel).
Figure 13. Comparison of NARX outputs and target joint angles in reduced space (single channel).
Sensors 21 02576 g013
Sensors 21 02576 g014 550
Figure 14. Performance of the proposed model referring to individual movements of fingers.
Figure 14. Performance of the proposed model referring to individual movements of fingers.
Sensors 21 02576 g014
Sensors 21 02576 g015 550
Figure 15. Comparison of NARX model output and the target in daily movements.
Figure 15. Comparison of NARX model output and the target in daily movements.
Sensors 21 02576 g015
Sensors 21 02576 g016 550
Figure 16. Mean square error of daily functional movements.
Figure 16. Mean square error of daily functional movements.
Sensors 21 02576 g016
Table
Table 1. Regression value of the proposed model.
Table 1. Regression value of the proposed model.
% of DataMSER
Training703.95420.989
Validation159.44330.985
Testing1510.02190.982
Table
Table 2. MSE of outputs of the NARX model and ordinary multilayer perception neural network (MLPNN) model over subjects.
Table 2. MSE of outputs of the NARX model and ordinary multilayer perception neural network (MLPNN) model over subjects.
NARXMLPNN
Subject 15.68754.483
Subject 210.95969.872
Subject 39.12262.162
Subject 43.88135.721
Subject 55.89147.635
Subject 610.95976.403
Subject 713.02752.895
Subject 832.22339.113
Subject 913.19647.868
Subject 1022.84360.540
Table
Table 3. Regression value of outputs of the NARX model and ordinary MLPNN model over subjects.
Table 3. Regression value of outputs of the NARX model and ordinary MLPNN model over subjects.
NARXMLPNN
Subject 10.98340.8054
Subject 20.97870.7618
Subject 30.97970.7776
Subject 40.98450.8187
Subject 50.98100.7723
Subject 60.97910.7592
Subject 70.97320.7930
Subject 80.93470.7742
Subject 90.97620.8379
Subject 100.97680.8787
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Gao, Z.; Tang, R.; Huang, Q.; He, J. A Multi-DoF Prosthetic Hand Finger Joint Controller for Wearable sEMG Sensors by Nonlinear Autoregressive Exogenous Model. Sensors 2021, 21, 2576. https://doi.org/10.3390/s21082576

AMA Style

Gao Z, Tang R, Huang Q, He J. A Multi-DoF Prosthetic Hand Finger Joint Controller for Wearable sEMG Sensors by Nonlinear Autoregressive Exogenous Model. Sensors. 2021; 21(8):2576. https://doi.org/10.3390/s21082576

Chicago/Turabian Style

Gao, Zhaolong, Rongyu Tang, Qiang Huang, and Jiping He. 2021. "A Multi-DoF Prosthetic Hand Finger Joint Controller for Wearable sEMG Sensors by Nonlinear Autoregressive Exogenous Model" Sensors 21, no. 8: 2576. https://doi.org/10.3390/s21082576

APA Style

Gao, Z., Tang, R., Huang, Q., & He, J. (2021). A Multi-DoF Prosthetic Hand Finger Joint Controller for Wearable sEMG Sensors by Nonlinear Autoregressive Exogenous Model. Sensors, 21(8), 2576. https://doi.org/10.3390/s21082576

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop