A Residual Temporal Convolutional with Attention Neural Network for Electromyogram-Based Hand Gesture Recognition

Electromyography (EMG)-based hand gesture recognition has become a pivotal technology in developing intuitive and responsive human-computer interaction (HMI) systems. In particular, prosthetic control represents a major application domain in biomedical engineering. Surface EMG signals, which reflect underlying muscle activity during contraction, offer a non-invasive and effective way to decode user gesture intent, particularly for amputees [1]. The process involves classifying muscle activation patterns into discrete hand or wrist movements, enabling intuitive control over assistive devices. However, the use of EMG signal classification presents several challenges due to the non-stationary nature of the signal, inter-subject variability, muscle fatigue, and noise introduced by sensors or motion artifacts. Therefore, accurate EMG signal classification requires models that can robustly extract temporal and spatial features from multi-channel time-series data especially in real-world scenarios where generalization across users and sessions is required.

In recent years, deep learning (DL) has exhibited impressive capability in biomedical signal processing. In the context of electromyography (EMG)-based hand gesture recognition, DL has demonstrated remarkable promise, offering the potential to build more accurate and adaptive human-machine interfaces (HMIs) [2, 3]. Unlike traditional machine learning algorithms that rely on handcrafted features, which often restrict their scalability and generalization to unseen data, deep neural networks can automatically extract hierarchical and temporal features directly from raw EMG data [4, 5]. CNNs have been widely applied for capturing spatial representations, while RNNs and LSTMs are used for sequential modeling. These capabilities make them well-suited for handling dynamic, high-dimensional time-series data such as EMG. More recently, hybrid approaches combining CNNs, LSTMs, or attention mechanisms have improved recognition accuracy, yet they often struggle with training efficiency, long-term temporal modeling, and generalization across subjects.

Temporal Convolutional Networks (TCNs) have emerged as a compelling alternative to recurrent architectures, thanks to their use of causal convolutions and dilation. These properties allow TCNs to model long-range temporal dependencies. Unlike standard CNNs, which are primarily designed for spatial feature extraction, TCNs utilize causal convolutions and dilation, enabling them to preserve sequence order while covering broad temporal contexts. This structure not only alleviates the vanishing gradient problem commonly associated with recurrent architectures but also allows for more efficient parallelization during training [6, 7]. Despite their promise, their application in EMG-based hand gesture recognition remains underexplored, particularity in combination with advanced techniques such as residual connections, attention mechanisms, and systematic data augmentation.

This study aims to address this gap by proposing a TCN-based model with residual connections and contextual attention for robust EMG-based hand gesture recognition. Unlike prior CNN or LSTM approaches, our model is designed to i) capture long-range temporal dependencies efficiently, ii) enhance discriminative focus through attention, and iii) mitigate overfitting via data augmentation. Our hypothesis is that this architecture can achieve superior accuracy and generalization compared to existing methods.

To validate our approach, the proposed model is trained and evaluated on a subset of the benchmark NinaProdataset, demonstrating competitive performance and promising generalization capabilities, which highlights its potential for real-time prosthetic control and human-machine interaction.

The remainder of this paper is organized as follows. Section 2 reviews related literature on deep learning for EMG-based gesture recognition. Section 3 presents the proposed methodology. Section 4 describes the experimental evaluation of the proposed approach, and Section 5 concludes the paper with a summary of contributions and directions for future improvement.

A range of approaches have been explored for EMG-based hand gesture recognition, spanning classic machine learning, deep learning, and hybrid architectures. This section highlights key works, emphasizing temporal modeling methods and those tailored to the NinaPro dataset. Atzori et al. [8] leveraged a LeNet-based CNN on NinaPro DB2, achieving 78–85% accuracy across multiple gestures with feature-based SVM classifiers, a solid baseline for later DL models. Geng et al. [9] introduced the concept of instantaneous sEMG imaging and applied a CNN plus SVM on 8 gestures captured via high-density sEMG imaging, attaining ~89.3% per-frame accuracy (and 99.0% after majority voting), illustrating the potential of single-frame classification. Du et al. [4] designed a custom LSTM-based model on NinaProDB2/DB3, achieving 85–90% accuracy on sequence recognition tasks, outperforming classical approaches.

While earlier studies demonstrated strong foundational performance using CNNs and LSTMs, more recent archi- tectures have adopted TCNs and attention mechanisms to improve both accuracy and efficiency. Tsinganos et al. [10] employed TCN for EMG-based hand gesture recognition using the NinaPro DB1 dataset, achieving around 89.76% accuracy and demonstrating the strength of TCNs in capturing temporal dependencies in muscle activity signals. Zanghieri et al. [11] proposed TEMPONetembedded TCN, A real-time embedded TCN model, achieving 93.7% accuracy on NinaPro enabled sessions with an ultra-low memory footprint.

Recent TCN and multi-stream architectures have further advanced the field. Rahimian et al. [12] proposed a lightweight TCN-attention model evaluated on 17 gesture classes, achieving 81.65% (300 ms windows) and 80.72% (200 ms windows), with ~12 $\times$ fewer parameters than benchmark models. A multi-stream deep architecture proposed by Shin et al. [13], integrating TCN, CNN, LSTM modules, and channel attention, was tested on NinaPro DB1 and DB9, attaining 94.3% and 98.96% accuracy, respectively, demonstrating the power of hybrid models for high-performance EMG classification.

Beyond these temporal models, recent research has introduced novel paradigms. Montazerin et al. [14] applied a Vision Transformer (ViT-HGR) to HD-sEMG, reporting ~84.6% accuracy on 65 gestures with a remarkably compact parameter count, illustrating the promise of transformer-based methods. Zhong et al. [15] developed a Spatio-Temporal Graph Convolutional Network (named STGCN-GR) for HD-sEMG-based human-machine interfaces, which explicitly models electrode topology and reached 91.07% accuracy on 65 gestures, outperforming CNN-based baselines. Xiang et al. [16] introduced SE-DenseNet with channel attention, achieving 85.93% and 82.39% accuracy on NinaPro DB2 and DB4, respectively, highlighting the effectiveness of attention in EMG recognition.

To synthesize these advancements, Table 1 provides a comparative overview of representative studies. It summarizes the datasets used (including number of gestures), core architectures, and reported performance metrics, offering a consolidated view of how different methods perform across diverse experimental contexts.

Although our work employs temporal convolutions similar to those used in related studies, our proposed architecture distinguishes itself by incorporating multiple residual causal convolution blocks and a contextual attention mechanism placed at the top of the network to enhance feature discrimination. Moreover, unlike prior works that focus on reducing the number of layers to achieve lightweight models, our design leverages deeper residual structures to improve representational capacity. Additionally, we apply suitable data augmentation techniques to enhance learning and promote better model generalization.

There exists a number of publicly available sEMG hand gesture datasets. These data sets vary in terms of acquisition device, number of subjects, types of gestures, number of gesture repetitions, etc. In this context, NinaProdatabase [2] is considered to be one of the most comprehensive resources, today comprising 9 separate data sets collected from 204 healthy subjects and 15 amputee subjects. NinaProcontains a set of upper limb electromyographic, kinematic and dynamic data to allow the public to test machine learning algorithms for controlling hand prostheses and conducting other related researches. The 9 datasets include multiple repetitions of at least 50 hand movements. The latter were recorded with different acquisition protocols and configurations, to allow the recording of several multi-modal signals. Despite the fact that NinaPro only includes data from 15 hand amputees, it has been shown that data from healthy subjects can also be used as a proxy measure for amputees. This result justifies the use of healthy subjects in order to reduce the stress and pain that can be caused in amputees [17].

NinaPro DB1 contains sEMG data captured from 10 channels, obtained at a sampling frequency of 100 Hz. In our experiments, we use data of 10 subjects from NinaPro DB1. For classification, we consider a total of 22 gestures provided in Figure 1. Each gesture was repeated 10 times. The dataset also includes a "Restimulus" label, which represents the same movements, but with improved time labels that better match when the movements really happened. The raw sEMG signals were preprocessed and structured into a consistent format for training and evaluating the model. The preprocessing includes segmenting the EMG signal according to the repetition and windowing. This process is performed for each subject separately.

Inspired by established methods in time-series literatures [18, 19], we apply the following multiple data augmentation techniques, which attempt to simulate real-world EMG signal variations during training:

• Jittering: adds adjustable Gaussian noise to match a target SNR, mimicking real-world EMG noise from sensors or ambient electrical interference.

• Channel Shuffling: circularly rotates the EMG channel order by a random offset ($\pm$ 2 channels). It models slight misplacement of electrodes, especially relevant for wearable EMG systems.

• Scaling: multiplies each channel by a random factor sampled from a normal distribution around 1 ($\sigma$ = 0.2). It simulates inter-session variability like sensor gain or changes in muscle contraction strength.

• Permutation: divides the signal into segments and randomly shuffles them. It encourages the model to focus on local temporal features rather than fixed global ordering.

As outlined in Figure 2, our proposed model, named Residual Temporal Convolutional Attention Neural Network (RTCAN), attempts to capture both short-term and long-term patterns in EMG signals. To this end, we use i) causal convolutions which do depend only on past data combined with ii) residual connections that let the signal jump over some layers to help train deeper networks, and iii) an attention mechanism to focus on important parts of the signal. The input of the model is a preprocessed EMG sample of size n $\times$ 10, corresponding to the number of time steps and sensor channels. The output of the model is a discrete value specifying the recognized gesture.

In the following, we discuss the details of each block of the proposed architecture.

The role of the Residual Causal Convolutional Blocks is to identify specific patterns over time for robust gesture classification. The basic building blocks are temporal convolution layers. In contrast to the usual convolution where the output at time t depends on past, present, and future inputs ($x(t-k / 2$) to $(t+k / 2$), $k$ is the filter size), in the temporal convolution, the output depends only on the past time steps. Another advantage of the temporal convolution is the use of dilated convolutions to increase their receptive field without drastically increasing the number of parameters.

where, $y(t)$ is the output at time step $t, f(i)$ is the filter of size $k, x(t-d . i)$ is the input sequence, and $d$ is the dilation factor. This last controls the convolution receptive field.

Residual layers are added to maintain gradient flow, and the overall transformation can be described by:

where, $H(x)$ is the output, $F(x)$ is the function learned by the skipped convolution layers, and $x$ is the input that gets passed through via the residual connection.

Our model comprises four causal convolution blocks strengthened by two residual connections. The first is from the output of the first causal convolution block to the input of the third one, and the second connection is from the output of the third causal convolution block to the input of the contextual attention block. Each causal convolution block has specific parameters as follows: [ 2, 2, 2, 1] and [[32, 32], [32, 32], [64, 64], [128]] for the number convolution layers and their corresponding number of filters, respectively. The ReLU activation function is applied to all convolution layers. To prevent model overfitting L2 regularization is applied to the kernel weights and each convolution is followed by a dropout layer. A summary of the causal convolutional blocks’ parameters is provided in Table 2.

The residual causal convolution blocks help the model learn how EMG signals change over time, while keeping the order of the signals, before being processed by the contextual attention.

The attention mechanism implemented in this work builds upon the hierarchical attention networks proposed by Yang et al. [20], adapting it for temporal EMG signal analysis. Unlike traditional sequence-to-sequence attention that operates between encoder and decoder states, this implementation processes a single temporal sequence to identify the most discriminative time steps for classification. Its mathematical formulation is as follows. Given an input sequence representation $\mathrm{X} \in \mathbb{R}^{B \times T \times F}$, where $B$ is the batch size, $T$ is the number of time steps, $F$ is features' dimension, the attention mechanism computes importance weights through three learned components:

Transformation Matrix (W): Projects the input features into the attention space:

where, $W \in \mathbb{R}^{F \times F}$ and $b \in \mathbb{R}^F$ are learnable parameters.

Context Vector (u): Learns a fixed query vector that determines which time steps are relevant:

where, $u \in \mathbb{R}^F$ is learned during training.

Normalized Attention Weights: Computed via softmax with masking support:

where, $m_{\mathrm{t}} \in\{0,1\}$ is the mask value for step $t$.

The final context vector is the weighted sum:

The top of the model is the classification layer implemented as a fully connected layer with softmax activation and its size the number of classes (recognized gestures).

The experimental setup defines the configuration encompassing the dataset, the model architecture, and the training strategy.

To ensure exclusive data samples between train and test, for each subject, data is portioned into disjoint sets based on repetitions. Specifically, for each subject signals from repetitions {1, 3, 4, 6, 8, 9, 10} are used for training and those from repetitions {2, 5, 7} are used for test. This setting allows fair model evaluation under unseen test data. For training, data augmentation is applied with a noise level of 25 dB SNR, along with magnitude warping and time warping, each set to 0.2. The batch size for training is 32, and samples are weighted based on their class frequency.

The training process uses the Adam optimizer with an initial learning rate of 0.001. A constant learning rate schedule is applied throughout the 100 training epochs. To promote model generalization on unseen data, we employ i) weight decay with a regularization strength of 0.0001; ii) a dropout rate of 0.4 after convolutions; iii) L2 regularization with a coefficient of 0.0005.

Inspired by related works, we perform a subject-specific evaluation. Thus, for each subject, we train and evaluate the model with his own data ensuring that the test and train data subsets are disjoint.

In Table 3, we report train and test times and classification accuracy for each subject. These results demonstrate that the model achieves consistent and robust accuracy, indicating its suitability for gesture recognition tasks using sEMG signals from NinaProDB1. Most of the accuracies are satisfactory, with a mean test accuracy over subjects of 95.31%. The worst model is that of Subject #4 with an accuracy of 89.06%. The best model is that of Subject #5 with a perfect test classification. Overall, the models generalize well to unseen gestures, as indicated by the small difference between train and test accuracies of each subject.

In average a model takes a bit less than 15 minutes (834.70s) for training and requires almost 0.43 seconds for recognizing a hand gesture during test. Moreover, our developed model is lightweight, with a total of 84.758 parameters, all of which are trainable, occupying just 331.09KB storage memory. This compact design is well-suited for embedding, enabling real-life applications such as controlling myoelectric prosthesis. To this end, however, deeper investigation should be conducted in this context.

To gain a deeper insight into the experimental results, we depict the training graphs for the best and worst performing subjects in Figure 3 and Figure 4 as well as their corresponding confusion matrices in Figure 5 and Figure 6. The learning graphs of the worst subject (Figure 3) demonstrate an overfitting, as depicted in the divergence between the train and validation curves for both the loss and the accuracy. In contrast, the learning graphs of the best subject (Figure 4) demonstrate the generalization of the model, as depicted by an almost perfect much between the train and validation curves.

Since the test accuracy of the best subject is 100%, a perfect confusion matrix is achieved as shown in Figure 5. The classification of a number of gestures is mismatched in case of the worst subject as shown in Figure 4, reducing the overall classification accuracy to 89.06%. Specifically, some EMG signals of five gestures (3, 7, 9, 16 and 17) were misclassified with different rates, while the remaining gestures are perfectly recognized.

Directly comparing EMG-based hand gesture recognition (HGR) models remains challenging due to differences in experimental settings, including the number of subjects, gesture classes, recording sessions, sensor configurations, and preprocessing pipelines. These factors can significantly influence reported performance, making absolute comparisons less straightforward. Despite these challenges, when evaluated against the most relevant works reviewed in Section 2, our approach achieves an average classification accuracy of 95.31% on the NinaPro DB1 dataset, which is competitive with and in several cases surpasses recent baselines.

The combination of Temporal Convolutional Networks with residual connections, attention mechanisms, and extensive data augmentation appears to be particularly effective in enhancing temporal feature modeling and generalization. In addition to accuracy gains, our framework demonstrates greater robustness across different subjects, as reflected by reduced variance in inter-subject performance compared to baselines such as CNN and RNN-based models. This stability is largely attributable to residual connections and extensive data augmentation, which mitigate overfitting and improve generalization, an essential property for practical deployment in prosthetic control and human-computer interaction.

In this paper, we explored the development of a deep learning framework tailored for surface EMG-based hand gesture recognition, with a particular focus on the use of Causal Convolutional, which present the advantage of its ability to model long-term dependencies, maintain temporal dependence over layers, and support efficient parallel training. We applied a set of preprocessing steps to structure the dataset for optimal learning and introduced our proposed architecture, RTCAN. RTCAN integrates residual causal blocks with contextual attention to enhance the extraction of meaningful EMG temporal features for robust hand gesture recognition. We evaluated our proposed approach on the NinaPro DB1 dataset, where the obtained results demonstrate strong classification performance, with most gestures correctly recognized across different subjects. In the future, we intend to conduct deeper analysis of the misclassified gestures and low performing subjects. Moreover, we will investigate subject independent evaluation of our proposed system. Extending the number of gestures and subjects will inform how our system scales. Another promising area is the implementation of the model as a real-time systems and testing its performance on embedded and low-power devices to determine its practicality for wearable prosthetic applications.