Efficient multimodal video information fusion for isolated sign language recognition using a shared Swin Transformer and dynamic gating

Sign language is the primary means of communication for many people with hearing impairments, and automatic sign language recognition plays an important role in improving information accessibility [1]. Isolated sign language recognition (ISLR), which identifies individual signs from short video clips, is a fundamental component of continuous sign language translation systems. It is also a challenging video information processing task because recognition depends on the coordinated interpretation of hand shape, motion, body posture, and facial information across successive frames. The Chinese Sign Language 500 (CSL-500) dataset is one of the largest publicly available datasets for Chinese ISLR and contains 500 commonly used words. Many of these words share similar hand shapes, motion trajectories, or facial expressions, making fine-grained visual discrimination difficult. A recent systematic review of 76 sign language recognition studies published between 2021 and 2026 identified inadequate multimodal fusion and limited cross-signer generalization as persistent challenges in this field [2].

Existing ISLR methods predominantly process red-green-blue (RGB) videos using three-dimensional convolutional networks, such as 3D Convolutional Networks (C3D) and Inflated 3D ConvNet (I3D) [3], [4], or video Transformers, such as TimeSformer [5] and Video Swin [6], to extract spatial and temporal representations. Although these methods have achieved competitive recognition performance, their decisions are based mainly on appearance information. As a result, they make limited use of geometric characteristics that may distinguish signs with similar two-dimensional appearances. For example, the signs for “thank you” and “sorry” may appear similar in RGB frames, even though their relative hand positions differ along the depth dimension. Depth information can therefore provide complementary evidence for differentiating visually similar signs. Its conventional acquisition, however, depends on dedicated red-green-blue-depth (RGB-D) sensors such as Kinect. These devices increase hardware cost and may perform poorly under strong outdoor illumination, limiting their use in portable and camera-based recognition systems. An alternative is therefore required to recover geometric information directly from existing RGB videos without changing the acquisition equipment.

Recent advances in monocular depth estimation provide a practical means of deriving geometric representations from RGB images. MiDaS [7] can estimate depth maps from individual RGB frames and can therefore be applied to previously recorded sign language videos without requiring additional sensors. Depth maps describe the relative spatial distribution of the hands and body, while surface-normal maps calculated from depth gradients represent local orientation changes around the fingers, joints, and palm. These two representations provide different but related forms of geometric information for fine-grained hand-shape analysis. Because both are generated from the original RGB frames, they should be regarded as complementary geometric representations rather than fully independent sensing modalities. Their use nevertheless expands the information available to the recognition model without modifying the original video acquisition process.

Effective integration of RGB, depth, and normal representations raises two main problems: parameter efficiency and information fusion. A direct approach is to train separate feature extraction backbones for the three inputs. However, three independent Swin-Tiny networks would contain approximately 85.8 M parameters, increasing computational demand and the risk of overfitting. A shared backbone substantially reduces the parameter count but may not preserve the characteristics specific to each representation unless suitable adaptation mechanisms are introduced. The fusion process presents a second difficulty. Concatenation and averaging assign fixed roles to different information sources and do not explicitly model the interactions among RGB appearance, depth structure, and surface orientation. Existing gating mechanisms [8] can assign adaptive weights, but they commonly operate without explicit cross-modal interaction and may become persistently dominated by a single representation. Such behaviour limits the use of complementary information and may weaken recognition performance for previously unseen signers.

These limitations show that effective ISLR requires more than the addition of geometric inputs. A suitable framework must encode heterogeneous visual representations efficiently, preserve representation-specific characteristics, model their interactions, and regulate their contributions during fusion. It must also capture temporal dependencies within video sequences and distinguish among sign categories that differ only in subtle spatial or motion characteristics. Addressing these requirements within a unified architecture is important for improving both recognition performance and computational efficiency.

To address these problems, this study proposes the Shared Swin with Gated Multimodal Fusion Network (SGNNet), an efficient multimodal video information processing framework for signer-independent ISLR. SGNNet derives complementary depth and surface-normal representations from RGB videos, encodes the three inputs using a shared Swin Transformer with lightweight representation-specific adapters, and models their interactions through dynamic cross-attention gated fusion (DCAGate). A temporal encoder is used to aggregate frame-level information, and a class-embedding classification head is introduced to support fine-grained discrimination among visually similar signs. The framework is evaluated on CSL-500 in terms of recognition accuracy, model size, graphics processing unit (GPU) memory consumption, recognition-stage throughput, convergence behaviour, and gate-weight distribution. The main contributions are summarized as follows:

(1) Complementary geometric information generation: MiDaS is used to estimate depth maps from RGB frames before model training, and Sobel operators are applied to the depth gradients to generate surface-normal maps. This process provides spatial and local orientation information without requiring external depth sensors or changes to the original acquisition equipment.

(2) Shared feature encoding with lightweight adapters: A single Swin-Tiny backbone is shared across the RGB, depth, and normal representations. Three lightweight multilayer perceptron (MLP) adapters project the shared features into representation-specific spaces, preserving differences among the input representations while avoiding the substantially larger parameter count associated with three independent Swin-Tiny backbones. The complete SGNNet model contains 30.89 M parameters.

(3) Dynamic cross-attention gated information fusion: Cross-attention is applied before gating to model interactions among the RGB, depth, and normal features. A learnable temperature coefficient and entropy regularization are incorporated to prevent persistent dominance by a single representation while retaining adaptive variation in the fusion weights.

(4) Class-embedding-based fine-grained recognition: A learnable class-embedding matrix and a margin-based class-embedding loss are introduced to increase the separation between visually similar sign categories and improve fine-grained classification performance.

Early ISLR methods predominantly combined hand-crafted descriptors, such as histograms of oriented gradients (HOG), histograms of optical flow (HOF), and scale-invariant feature transform (SIFT), with hidden Markov models or support vector machines. With the development of deep learning, three-dimensional convolutional neural networks, including C3D, I3D, and P3D, became widely used for extracting spatial and temporal information from sign language videos. Attention mechanisms were subsequently incorporated into two-stream 3D convolutional architectures to improve the representation of dynamic gestures [9]. More recently, video Transformers, including TimeSformer and Video Swin Transformer, have achieved strong results on several video recognition benchmarks because of their capacity to model long-range spatial and temporal dependencies. Pretrained vision Transformers have also shown good generalization in ISLR [10]. Liu et al. developed DeepSignV1 [11], a vision Transformer-based ISLR model that used pretraining and architectural design to outperform conventional 3D convolutional approaches on benchmark datasets. CrossViViT [12] combined cross-attention with a video vision Transformer for Vietnamese sign language recognition, but its input remained limited to RGB videos.

Despite this progress, most existing approaches derive their decisions primarily from RGB appearance information. Their performance may therefore be affected by illumination changes, background variation, and visual similarities among signs. More importantly, RGB-only methods do not explicitly represent three-dimensional hand geometry. Depth and surface-orientation information can provide complementary cues for distinguishing signs with similar two-dimensional appearances but different spatial configurations. The incorporation of geometric representations, including depth and surface-normal maps, is therefore a useful direction for improving fine-grained sign language recognition.

Several studies have combined RGB and depth information acquired using RGB-D cameras [13]. Molchanov et al. [14] used depth maps for hand segmentation and dynamic gesture recognition. For Chinese sign language, Li et al. [15] developed an end-to-end multimodal fusion system for continuous dynamic gesture recognition. An RGB–skeleton fusion framework [16] was subsequently investigated for Chinese sign language recognition, although it depended on an external pose estimator and was evaluated on a subset containing only 50 classes. These approaches show the value of geometric information, but their dependence on dedicated depth sensors or external pose-estimation systems may restrict deployment in camera-based recognition environments.

Monocular depth estimation provides an alternative means of obtaining geometric information from existing RGB videos. An off-the-shelf depth estimator was used to support video-based gesture recognition [17], while Sarhan [18] generated pseudo-depth representations from RGB data to improve sign language recognition. These studies relied mainly on depth information and did not examine surface-normal representations, which describe local orientation changes around the palm, fingers, and joints. Gao et al. [19] proposed CAMFuse, a cross-attention fusion network that combined RGB and skeleton representations for video sign language recognition. The model achieved an accuracy of 96.18% on the AUTSL dataset by modelling spatial and temporal interactions between the two inputs. FusionEnsemble-Net [20] subsequently combined RGB videos with radar data through attention-based ensemble learning, but the method required an additional radar sensor.

Unlike sensor-dependent approaches, the present study generates both depth and surface-normal representations from the original RGB frames. Depth maps describe relative spatial structure, whereas surface-normal maps characterize local orientation. The two representations are not independent sensing modalities because both are derived from RGB data. They nevertheless provide complementary geometric descriptions without requiring additional acquisition hardware. This design supports richer video information representation while retaining compatibility with conventional RGB video datasets and camera-based recognition systems.

Multimodal fusion is generally implemented at the input, feature, or decision level. Input-level fusion combines raw data before feature extraction, whereas decision-level fusion combines the outputs of separately trained models. Feature-level fusion allows interactions among intermediate representations and is therefore widely used in multimodal recognition. Attention mechanisms and gating networks are two common approaches to feature-level fusion. The Gated Multimodal Unit assigns adaptive weights to different representations through a gating network, but it does not explicitly model cross-modal interactions. Multimodal Transformers such as MulT [21] use cross-modal attention to capture these interactions, although their computational cost can be substantial. MB-SLR [22] introduced bidirectional visual–pose alternating attention for multimodal sign language recognition, but it relied on external pose estimation and did not directly control persistent dominance by one representation.

Gating-based fusion may also experience modality collapse [23]. During training, the weight assigned to one representation may approach one while the remaining weights approach zero. The fusion module then behaves largely as a unimodal model and makes limited use of complementary information. This problem is particularly relevant when the input representations differ in scale, information density, or learning difficulty.

The proposed DCAGate module addresses this problem in two ways:

(1) Lightweight cross-attention is applied before gating to model interactions among the RGB, depth, and normal representations.

(2) An entropy regularization term is added to discourage persistent dominance by a single representation and maintain a more balanced distribution of gate weights.

This design separates cross-representation interaction from contribution regulation. Cross-attention exchanges information among the representations, while the gating mechanism determines their relative contributions at each time step.

Semantic information associated with category names has been used in zero-shot learning and fine-grained visual classification. Frome et al. introduced DeViSE, which represented categories using Word2Vec vectors and performed zero-shot classification through visual–semantic alignment [24]. Contrastive language–image pretraining, represented by CLIP, subsequently demonstrated strong zero-shot visual recognition by aligning image and text representations in a shared feature space [25]. These methods depend on externally generated linguistic representations and are mainly designed to transfer semantic relationships across categories.

Learnable class-embedding matrices provide a different approach. Instead of fixing category representations using a pretrained language model, class embeddings are optimized together with the visual recognition network and act as adaptable class prototypes. This approach allows the category representations to reflect the visual structure of the training data, which is useful when classes differ in subtle hand shapes, movements, or spatial configurations.

Following this approach, the present study uses a learnable class-embedding matrix and calculates cosine similarity between the global video feature and each class embedding as a supplement to the direct classification logits. A margin-based class-embedding loss requires the similarity score of the correct class to exceed the second-highest score by at least 0.3. This constraint increases the separation between the target category and its closest competing category, thereby supporting fine-grained discrimination among visually similar signs.

The overall architecture of SGNNet is illustrated in Figure 1. The framework consists of five sequential components: complementary representation generation, shared feature encoding, DCAGate, temporal encoding, and class-embedding-based classification.

Given a sign language video clip containing \(T\) frames, the corresponding RGB, depth, and normal image sequences are denoted as \(\{R_t, D_t, N_t\}\). The \(T\) frames from each representation are then processed by the shared Swin-Tiny backbone and the corresponding representation-specific adapter, producing three 128-dimensional feature sequences \(\mathbf{R}, \mathbf{D}, \mathbf{N} \in \mathbb{R}^{B \times T \times 128}\), where \(B\) denotes the batch size. DCAGate modelled interactions among the three feature sequences through cross-attention and assigned representation weights at each time step, producing the fused sequence $F_{\text{out}}$. The temporal encoder, composed of a bidirectional long short-term memory network and multi-head self-attention, modelled local and long-range temporal dependencies. Attention pooling subsequently aggregated the temporal sequence into the global video feature $v_{\text {global }}$. Finally, the class-embedding classification head combined the direct classification logits with the class-embedding similarity scores to produce a probability distribution over the 500 sign classes. The complete network was trained end to end through joint optimization of the cross-entropy loss, gate entropy loss, and class-embedding margin loss.

The MiDaS-family DPT-Large model [26], built on a Vision Transformer (ViT)-L/16 backbone, was selected as the monocular depth estimation model because it can capture both global context and local structural information. Its pretrained weights were kept fixed, and the proposed adjustments were applied during inference without retraining the model.

Considering that hand actions require both global pose consistency and local details of finger joints, we adopt a dual-scale input strategy. The input RGB frames are resized to 384$\times$384 and 512$\times$512, with the former focusing on global structure and the latter preserving local texture. The images at both scales are fed into the DPT-Large model (ViT-L/16 backbone, pre-trained weights fixed) for depth inference, outputting inverse depth maps which are then inverted to “near bright, far dark” grayscale maps. Subsequently, bicubic interpolation restores both scale depth maps to the original image resolution, and they are fused with a weighted formula:

The weights were determined by searching with a step of 0.1 on the validation set, ensuring that global consistency is maintained while enhancing hand details.

The fused depth map still suffers from insufficient contrast, residual noise, and edge misalignment. To address these issues, we construct a four-step post-processing pipeline:

(1) Contrast-limited adaptive histogram equalization (CLAHE) contrast enhancement ($\mathrm{clipLimit}=4.5$, $\mathrm{tileGridSize}=(8,8)$): to improve hand background discrimination and enhance finger-joint textures.

(2) Bilateral filtering ($d=5$, $\sigma_{\mathrm{color}}=50$, $\sigma_{\mathrm{space}}=50$): to smooth noise while preserving edges.

(3) Guided filtering for edge alignment: using the original RGB image as a guide ($r=10$, $\epsilon=0.01$). To precisely align depth edges with RGB edges, eliminating misalignment artifacts.

(4) Laplacian sharpening ($3\times3$ kernel): to enhance finger contours and joint details, making the depth map clearer.

After the above four steps, the clarity of the depth map is improved. Normal maps are then computed from depth gradients using Sobel operators to obtain gradients in the $x$ and $y$ directions, and the normal vector is constructed as:

Each component of the normal vector is linearly mapped from $[-1,1]$ to $[ 0,255]$ :

The normal map encodes the surface orientation information hidden in the depth map as an RGB image. Flat palm areas have normals approximately pointing toward the viewer (appearing blue), while finger edges and joints show sharp variations in normal direction (appearing as red-yellow gradients). This representation effectively captures the 3D structure of the hand surface and provides rich geometric cues for subsequent feature extraction. The normal map is also resized to 224$\times$224.

To intuitively demonstrate the effectiveness of our depth and normal map generation methods, Figure 2 shows a comparison of the three modalities for a typical gesture sample from the CSL-500 dataset. The RGB image shows that there is some color similarity between the hand and background, making hand segmentation difficult by appearance alone. The enhanced depth map generated by our method, after dual-scale fusion and the four step post processing pipeline, presents clear depth differences between finger contours (especially at fingertips and joints) and the background, with hand edges precisely aligned to the RGB image. The normal map derived from depth gradients encodes surface orientation information; flat palm areas appear blue, while finger edges and joints show drastic changes in normal direction (red-yellow gradients), effectively encoding the 3D orientation of the hand surface.

The above three-representation configuration comparison indicates that the depth map provides distance information that RGB cannot offer, and the normal map further supplements surface orientation cues. Together with RGB appearance information, they form complementary geometric features for subsequent multimodal sign language recognition.

Swin-Tiny was adopted as the shared backbone for the three input representations to balance feature extraction capacity and parameter efficiency. The backbone was pretrained on ImageNet-1K and produced a 768-dimensional feature vector for each input frame. Given the limited number of samples per class, the parameters of the first two stages were frozen to retain the pretrained visual representations and reduce the risk of overfitting, whereas the final two stages were fine-tuned on the CSL-500 training set. For each modality image $x_m(m \in\{$ RGB,Depth, Normal$\})$:

Then a representation-specific adapter reduces the dimensionality to $d=128$:

The three adapters are not shared, but their total parameter count is only about 0.06 M . Finally, the threerepresentation configuration feature sequences $\mathbf{R}, \mathbf{D}, \mathbf{N} \in \mathbb{R}^{B \times T \times 128}$, are obtained, where B is the batch size.

Parameter analysis: Swin-Tiny has 28.6 M parameters, and the three adapters add 0.06 M, totaling 28.66 M. The independent three-backbone scheme would require $3 \times 28.6 = 85.8$ M, a reduction of 66%. Among trainable parameters, the last two Swin layers have about 5 M, and the adapters and subsequent modules add about 0.5 M, giving a total trainable parameter count of about 5.5 M, which is efficient for training. This design balances the retention of pre-trained knowledge and representation-specific adaptation, avoiding the huge parameter count of independent backbones while maintaining differentiated representations for each modality. The architecture of the shared backbone and representation-specific adapter is illustrated in Figure 3.

Taking RGB as an example, cross-attention is computed between the RGB features and the Depth and Normal features. The features of each modality are linearly projected into queries, keys, and values as follows:

Multi head attention (number of heads $h=4$, each head dimension $d / h=32$) computes:

The enhanced RGB feature is:

Similarly, $\tilde{\mathrm{D}}$ and $\tilde{\mathrm{N}}$ are obtained. The coefficient 0.5 balances the self and enhanced features. The choice $h=4$ was based on a hyperparameter search on the validation set, where 4 heads achieved the best balance between capturing inter-modal interaction and computational efficiency.

The three enhanced features are then concatenated as follows:

The gating network is a three-layer MLP: $384 \xrightarrow{\mathrm{GELU},\,\mathrm{Dropout}(0.1)} 128 \xrightarrow{\mathrm{GELU},\,\mathrm{Dropout}(0.1)} 64 \rightarrow 3$. This progressive dimensionality reduction structure retains sufficient representation capability while reducing parameters. The output logits are passed through a temperature-scaled softmax to obtain the weights:

where, $\tau_0$ is a learnable parameter initialized to 1.5 . The temperature $\tau$ controls the sharpness of the distribution: larger $\tau$ leads to more uniform weights, and smaller $\tau$ leads to sharper weights. Finally, the following constraint is satisfied:

The fused feature is:

where, $a=0.15$ is the residual coefficient, experimentally verified to preserve an information pathway for low-weight modalities without interfering with the adaptive gating. The residual term ensures that even if a modality has a very low weight, its original information can still influence the final feature via the residual path. Finally, we apply projection, Dropout, and LayerNorm, followed by a skip connection:

To prevent the gate from collapsing into a one-hot distribution, we introduce a gate entropy loss:

This loss maximizes entropy, encouraging uniform weights. When $g_m=1 / 3$ for all three modalities, the entropy reaches its theoretical maximum of $\ln 3 \approx 1.099$.

To summarize the overall procedure, the complete pipeline of the proposed DCAGate module is illustrated in Figure 4.

A cascaded bidirectional long short-term memory (Bi-LSTM) and self-attention architecture was adopted for temporal encoding. The Bi-LSTM captured local temporal dependencies, while the subsequent self-attention module modelled long-range interactions across the video sequence. The fused feature sequence was first processed by a two-layer Bi-LSTM with 64 hidden units in each direction. Concatenating the forward and backward outputs produced a 128-dimensional feature at each time step. A dropout rate of 0.3 was applied between the recurrent layers to reduce overfitting. Subsequently, the Bi-LSTM output $H$ is passed to a multi-head self-attention module with 4 attention heads, model dimension $d_{\text {model}}=128$, and dropout rate 0.3, which models long range dependencies through global interactions. Meanwhile, the self-attention output output is combined with a residual connection and layer normalization (LayerNorm) to ensure training stability. Finally, an attention pooling mechanism based on a learnable query vector is adopted, where the temporal attention weights are computed as in Eq. (20) and the weighted sum is performed as in Eq. (21) to obtain the final global feature representation $v_{\text {global}}$:

where, $H \in \mathbb{R}^{128 \times T}$ is the Bi-LSTM output arranged along the time steps, $w \in \mathbb{R}^{128}$ is a learnable query vector, $a_t$ is the attention weight for the $t$-th frame, and $v_{\text {global }} \in \mathbb{R}^{128}$ is the resulting global feature vector.

The semantic-aware classification head consists of two branches: a direct classification branch and a class-embedding branch. Let the number of classes be $C=500$. The direct branch is a two-layer MLP:

The direct classification branch outputs $\mathbf{I}_{\mathrm{dir}} \in \mathbb{R}^{C}$. The semantic branch maintains a learnable class-embedding matrix $E_{\mathrm{cls}} \in \mathbb{R}^{C \times 128}$. The global feature $v_{\mathrm{global}}$ is projected through a linear layer $(128 \rightarrow 128)$ to obtain $v_{\mathrm{sem}}$, and the cosine similarity between $v_{\mathrm{sem}}$ and the class embeddings is computed as follows:

where, $\tau_{\text {sem }}$ is learnable, initialized to 1.0 . The final logits are $l=l_{\mathrm{dir}}+0.5 l_{\mathrm{sem}}$.

The overall loss is formulated as in Eq. (24), where $\mathcal{L}_{cls}$ is the cross-entropy loss, and $\mathcal{L}_{gate}$ and $\mathcal{L}_{sem}$ are auxiliary losses with weights 0.3 and 0.1, respectively. The model is trained using AdamW with layer-wise learning rates ($1 \times 10^{-5}$ for Swin backbone, $2 \times 10^{-4}$ for other modules), cosine annealing restart scheduler, mixed precision, batch size 2, sequence length 20, and early-stopping patience 7.

Our experiments were conducted on hardware comprising an Intel Core i7-10875H CPU, an NVIDIA GeForce RTX 2060 GPU (6 GB VRAM), and 16 GB RAM. The software environment was Ubuntu 20.04 LTS, Python 3.8, PyTorch 1.12.0, torchvision 0.13, and the official MiDaS implementation.

The dataset used is CSL-500 [27], one of the largest isolated-word datasets in the Chinese sign language domain. It contains 500 common sign vocabulary items and approximately 25,000 video clips (about 50 per class), recorded by 20 deaf students of different genders and ages. The videos have a resolution of 640 $\times$ 480, duration of 1 to 3 seconds, and frame rate of 25 frames per second (fps). We split the dataset into training (20,000), validation (2,500), and test (2,500) sets with a ratio of 8:1:1, ensuring that videos from the same signer do not appear across different splits to guarantee signer independence. Depth maps were pre-computed using the MiDaS DPT-Large model (dual-scale fusion, saved as .npy files), and normal maps were generated from the depth maps using Sobel operators. During training, the data were loaded from cache to accelerate loading.

The following metrics were adopted to evaluate model performance:

(1) Top-1 accuracy (%): the proportion of correctly classified samples among the total test set, serving as the primary evaluation metric.

(2) Parameter count (M): the total number of model parameters, in millions, reflecting the model size.

(3) GPU memory usage (GB): the maximum GPU memory consumption during training with a batch size of 2, reflecting the computational resource demand.

(4) Recognition network throughput (fps): the number of video frames processed per second by the recognition network (Swin backbone + DCAGate + temporal encoder) on a single RTX 2060 GPU, measured with a sequence length of 20 frames.

To validate the effectiveness of SGNNet, we compared it with state-of-the-art methods reported on the CSL-500 dataset. The results are shown in Table 1.

From Table 1, the skeleton based SLR Net achieves the highest accuracy of 98.08%, demonstrating that precise pose information boosts recognition performance. However, such methods rely on external pose estimators (e.g., OpenPose), which suffer from degraded accuracy under hand occlusion or complex lighting, limiting their practicality.

Compared with RGB video based methods Lightweight 3D CNNs + Transformer (95.12%) and ST-GCN (94.40%), our SGNNet with the introduction of depth and normal geometric information, reaches 96.20%, confirming the effectiveness of multimodal geometric cues. STGCN LSTM fuses phonological features to achieve 95.20%, but it requires prior knowledge of sign language linguistics, restricting its generalization ability.

Overall, SGNNet achieves 96.2% accuracy for signer-independent recognition on CSL-500, ranking among the top-performing methods. These results demonstrate the effectiveness of incorporating complementary geometric representations for signer-independent sign language recognition.

To verify the effectiveness of the proposed shared Swin backbone architecture, two baseline models were designed for comparison. The first is a lightweight three-representation configuration model with independent convolutional neural network (CNN) backbones, in which each modality is processed by a separate lightweight convolutional network. Each backbone has 3 convolutional layers (channels: input $\rightarrow 16 \rightarrow 32 \rightarrow 64$, kernel size 3 $\times$ 3, stride 2), followed by adaptive average pooling and flattening to output a 64-dimensional feature vector. All three backbones are randomly initialized (without pre-training) and their features are extracted in parallel before being fed into the dynamic gated fusion module and finally a classification head. This model represents a multimodal baseline with no prior knowledge, trained purely from scratch. The second baseline uses a ResNet18 backbone: the RGB branch employs a pre-trained ResNet18 on ImageNet (fine-tuning layer4, output dimension 512); the depth and normal branches each use a lightweight CNN (output dimension 128). The three branches extract features in parallel, followed by weighted fusion via the dynamic gated fusion module. This model represents a traditional multimodal independent encoding scheme. Our model uses the shared Swin-Tiny backbone with three lightweight adapters. The shared Swin-Tiny (pre-trained, first 6 layers frozen) extracts generic visual features, and then three lightweight adapters ($768 \rightarrow 128$) generate representation-specific embeddings. All models were trained under the same data split (train/validation/test = 8:1:1), with input image resolution 224 $\times$ 224 and sequence length 20 frames.

As shown in Table 2, the lightweight independent CNN achieves only 77.0% accuracy, indicating that randomly initialized lightweight networks have severely insufficient representation capability for the 500-class fine-grained classification task, validating the importance of pre-trained models. ResNet18 backbone reaches 92.4% with 14.67 M parameters.

Notably, although the shared Swin backbone has more parameters (30.89 M) than ResNet18 (14.67 M), its GPU memory usage is lower (2.5 GB vs. 3.8 GB). The lower GPU memory usage may be attributed to the shared-backbone implementation and the avoidance of multiple independent backbone calls. The window-based attention mechanism of Swin Transformer may also contribute to computational efficiency. However, the observed memory difference reflects the overall implementation rather than a single architectural factor. Moreover, the recognition network throughput of the shared Swin backbone (115 fps) is better than that of ResNet18 (92 fps), indicating its efficiency advantages for recognition after the geometric representations have been pre-computed. The reported recognition network throughput and GPU memory usage refer to the recognition network only and exclude offline depth and normal map generation.

To validate the effectiveness of the proposed multimodal and dynamic gated fusion mechanisms, two models with different modality configurations were designed for comparison. Model 1 employs a ResNet18 backbone, whereas Model 2 employs the proposed shared Swin backbone.

From Table 3, in the single-modality comparisons, RGB alone performs best (90.2%), while depth alone achieves a limited accuracy (81.5%). However, when depth is combined with RGB, the accuracy increases from 90.2% to 94.1%, a gain of 3.9 percentage points, demonstrating the complementary importance of depth information to visual features. On this basis, our full model further achieves 96.2%, an improvement of 2.1 percentage points over RGB + Depth (94.1%), verifying that the surface orientation information carried by the normal map derived from depth can effectively enhance fine-grained hand-shape discrimination. Under the same three-representation configuration condition, our model (96.2%) outperforms Model 1’s three-representation configuration result (92.4%), indicating that the shared Swin backbone with lightweight adapters offers advantages in both parameter efficiency and feature representation. The RGB-only Swin model achieves 90.2% accuracy, slightly below the 90.6% obtained by the ResNet18-based model. Given the small difference, this result should not be interpreted as evidence of inferior representation capability. The advantage of the shared Swin architecture becomes more apparent when complementary geometric representations are incorporated.

To further analyze the training behavior under different modality configurations, Figure 5 shows the training loss and accuracy curves for the full model and the dual-modal model.

From Figure 5a and Figure 5b, the full model shows a training loss steadily decreasing from about 2.45 to 0.27, a validation loss converging to about 0.28, and a final validation accuracy of 96.2%. In comparison, Figure 5c and Figure 5d show that the dual-modal model (RGB + Depth) achieves a final validation accuracy of 94.1%.

The full model improves by 2.1 percentage points over the dual-modal model, confirming that the surface orientation information provided by the normal map effectively enhances fine-grained hand-shape discrimination. Notably, the full model has a slightly slower convergence speed in the early epochs (first 10 epochs) compared to the dual-modal model, which may be due to the increased input dimensions after incorporating the normal map, requiring more iterations to fully fuse the three modalities. As training proceeds, the full model exhibits a superior convergence upper bound.

Moreover, neither model exhibits obvious overfitting during training (the gap between training and validation curves is small), indicating that our adopted early-stopping strategy is effective.

To verify the gating behavior of the DCAGate module under different modality combinations and its stability during training, we analyze the gating mechanism from two aspects: weight distribution balance and training dynamics.

The gate weights for the dual-modal (RGB + Depth) and trimodal (RGB + Depth + Normal) configurations were extracted for visual comparison, as shown in Figure 6.

From Figure 6, the three modality weights fluctuate around a balanced distribution rather than collapsing toward a single modality. From Figure 6a, in the triple-modal configuration, the weights of RGB, Depth, and Normal fluctuate slightly around the uniform reference line of 0.33, with none showing degeneration toward 1 or 0. From Figure 6b, in the dual-modal configuration, the weights of RGB and Depth fluctuate around the uniform reference line of 0.50, with local variations across frames. These local variations indicate that the gate retains adaptive weighting while entropy regularization prevents persistent dominance by one modality.

The above results indicate that:

(1) Regardless of whether the number of modalities is 2 or 3, DCAGate maintains the modality weights near a balanced distribution with local fluctuations, demonstrating that the gating mechanism retains adaptive weighting while having a stable regularization effect.

(2) The addition of the surface-normal representation does not disrupt the original balanced fusion state, further verifying the robustness of entropy regularization across different modality combinations.

(3) Combined with the quantitative results in Table 3 (dual-modal 94.1% → triple-modal 96.2%), the weight balance and classification accuracy improve simultaneously, indicating that DCAGate effectively enhances model performance while ensuring full multimodal fusion.

To further verify the effectiveness of entropy regularization throughout the training process, the gate entropy and maximum gate weight were recorded across training epochs for the trimodal configuration, as shown in Figure 7.

From Figure 7a, the gate entropy gradually rises from an initial value of about 0.6 and stabilizes in the range of 0.95 to 1.05, close to the theoretical maximum $\ln 3 \approx 1.099$, indicating that the weights of RGB, depth, and normal remain uniformly distributed throughout the entire training process, and the entropy regularization successfully prevents the gate from degenerating into single-modal dependence.

From Figure 7b, in the triple-modal configuration, the maximum gate weight remains stable at approximately 0.35 throughout training, well below the healthy threshold of 0.5 and the degeneration alert line of 0.9, further confirming that DCAGate does not suffer from modality collapse.

These results demonstrate that:

(1) The entropy regularization loss remains effective throughout training, with gate entropy stabilizing near the theoretical maximum.

(2) In the triple-modal configuration, the maximum gate weight stabilizes at approximately 0.35, and in the dual-modal configuration it stabilizes at approximately 0.50. Both remain well below the predefined degeneration threshold of 0.9, indicating that the model does not over-rely on any single modality.

(3) Combined with the weight distribution snapshots in Figure 6, these results show that DCAGate maintains balanced and stable weights across the RGB, Depth, and Normal modalities throughout training, without favoring any single modality. This finding indicates that entropy regularization effectively mitigates modality collapse.

This study developed SGNNet, a multimodal video information processing framework for signer-independent ISLR. Depth maps were estimated from RGB frames using DPT-Large with dual-scale inputs, and a post-processing procedure combining contrast-limited adaptive histogram equalization, bilateral filtering, guided filtering, and Laplacian sharpening was applied. Surface-normal maps were subsequently calculated from the processed depth gradients to provide local orientation information. A shared Swin Transformer with three lightweight adapters was used to encode the RGB, depth, and surface-normal representations. Cross-representation interactions were modelled through DCAGate, while a Bi-LSTM, self-attention, and class-embedding classification head were used for temporal modelling and fine-grained classification.

On the CSL-500 dataset, SGNNet achieved a Top-1 accuracy of 96.2% under the adopted signer-independent evaluation protocol. The complete model contained 30.89 M parameters, and the recognition network processed 115 frames per second with a peak GPU memory consumption of 2.5 GB. Compared with the independent-branch baseline, SGNNet increased accuracy by 3.8 percentage points, reduced GPU memory consumption by 34%, and increased recognition-stage throughput by 25%. The gate-weight analysis showed that the contributions of the three representations remained relatively balanced across the evaluated configurations, with no persistent dominance by a single representation. These results indicate that depth and surface-normal representations derived from RGB videos can provide useful geometric information for ISLR and that shared feature encoding can limit the computational cost of processing multiple representations.

The reported throughput and memory measurements apply only to the recognition network and exclude the offline generation of depth and surface-normal maps. In addition, the depth and surface-normal inputs were derived from RGB frames rather than acquired from independent sensors, and the evaluation was limited to CSL-500. Further work should therefore examine the method on additional sign language datasets, measure complete end-to-end processing time, and evaluate performance under changes in illumination, background, viewpoint, and hand occlusion. Extending the framework to continuous sign language translation, jointly training depth estimation and recognition, and incorporating hand-keypoint information are also relevant directions for future study.