Hybrid Convolutional Neural Network-Gated Recurrent Unit-Attention and Autoencoder Framework for Zero-Day Network Attack Detection: A Computational Approach with Experimental Evaluation on CICIDS2017

Signature-based network intrusion detection systems (NIDS) cannot detect zero-day vulnerabilities–unknown flaws exploited before patches exist [1]. Anomaly-based approaches address this by identifying novel attacks as deviations from learned normal behavior; autoencoders (AEs) detect zero-day attacks without labeled data but suffer elevated false-positive rates that impede security operations center (SOC) deployment [2], [3]. Convolutional Neural Network (CNN)-Gated Recurrent Unit (GRU) hybrids provide effective temporal-spatial feature extraction [4], while attention mechanisms further improve the model's ability to focus on relevant traffic patterns and generalize to unseen attack but fail on out-of-distribution events [5], while hybrid AE-supervised fusion consistently outperforms single-component systems [6], [7]. Recent researches confirm that deep learning architectures excel at complex hidden-pattern detection [8], [9], and SHapley Additive exPlanations (SHAP)-based eXplainable Artificial Intelligence (XAI) translates black-box decisions into interpretable feature attributions for analysts [10].

This paper proposes a hybrid CNN-GRU-Attention and convolutional AE framework for zero-day network attack detection on Canadian Institute for Cybersecurity Intrusion Detection System (CICIDS)2017, with three design contributions: (1) strict data-leakage prevention through temporal-logical dataset separation [11]; (2) deterministic reshaping of 64 selected features into 8 $\times$ 8 spatial images enabling CNN processing without manual feature engineering [12]; and (3) SHAP and reconstruction error explainability [10].

Elshewey et al. [4] demonstrate that CNN layers for spatial feature extraction combined with GRU layers for temporal modeling achieves near-perfect Distributed Denial-of-Service (DDoS) classification, establishing the core insight motivating our architecture. Alashjaee [5] adds scaled dot-product self-attention to a CNN-Long Short-Term Memory (LSTM) pipeline, improving generalization to unseen patterns. Pearson–Spearman–Kendall (PSK)-based feature selection confirms that correlation-aware dimensionality reduction preserves accuracy while reducing computational load [13], [14].

Xue et al. [6] propose Hybrid AE–Hybrid Resnet-LSTM (HAE-HRL), fusing an AE with a hybrid LSTM-CNN residual network to capture both packet-level and flow-level statistics simultaneously. Torabi et al. [3] introduce feature-wise reconstruction error thresholds, outperforming scalar-threshold alternatives and establishing feature-level anomaly scoring as the preferred zero-day detection technique.

Babaey and Faragardi [2] train LSTM, GRU, and stacked AE ensembles on normal traffic, achieving low false positive rate (FPR) on zero-day web attacks. Megantara and Ahmad [1] combine Decision Tree feature elimination with Local Outlier Factor (LOF) filtering for strong Denial-of-Service (DoS) detection. Hybrid AE-supervised fusion consistently outperforms single-method systems [7], [15], supporting the hybrid fusion strategy.

Rigorous evaluation requires fitting all preprocessing exclusively on training data and transforming validation/test sets using training statistics [11]. Reshaping flow features into 2D spatial images enables CNN processing without manual feature engineering [12]. SHAP-based explainability enables analysts to trace decisions to specific features [10]. Baselines include Isolation Forest [16], LOF [17], One-Class Support Vector Machine (SVM) [18], and Elliptic Envelope [19].

Existing hybrid frameworks lack rigorous leakage prevention, use arbitrary feature engineering, and offer limited interpretability of component interactions. This work addresses these gaps through: strict temporal-logical data separation; 64-feature selection reshaped into 8 $\times$ 8 images; OR-logic fusion with per-pathway attribution; and SHAP explainability with mechanistic decomposition of detection pathways.

We integrate a binary CNN-GRU-Attention classifier with a skip-connection AE, fused via logical OR: a sample is flagged as zero-day if the CNN anomaly score exceeds $\theta_s$ or reconstruction error exceeds $\theta_a$.

Five phases: (1) Score-based Label Separation and Ordering (SLSO) data splitting; (2) feature selection and 8 $\times$ 8 reshaping on $D_{\text{train}}$; (3) parallel CNN-GRU-Attention and AE training; (4) percentile threshold calibration on $D_{\text{val}}$; (5) OR-fusion inference with SHAP analysis. The component architectures are shown in Figure 1; the full pipeline and hybrid inference logic are shown in Figure 2.

To ensure unbiased model evaluation and prevent data leakage, we employ a rigorous splitting strategy with three distinct datasets:

Zero-day dataset. Each attack class $c$ receives a detectability score $S(c)$ (Eq. (1)) measuring its feature-space separation from the known-class manifold relative to its intra-class spread. Classes with $S(c) > 1.0$ are assigned to the zero-day hold-out set: heartbleed (74.6), DoS Slowhttptest (5.09), and infiltration (1.43). Classes with low scores–bot (0.41), DoS Slowloris (0.74), Web Attack–Structured Query Language (SQL) Injection (0.88)–are retained as known, as their feature distributions overlap substantially with already-known attacks; Portscan is excluded due to its degenerate feature structure. The zero-day set is isolated before any normalisation or feature selection, ensuring complete information isolation.

where, $\mu_c$ is the centroid of class $c$ in the CNN feature space, $d_{\cos}$ denotes cosine distance, $\sigma_c$ is the mean intra-class cosine spread (compactness), and $\varepsilon = 10^{-9}$ prevents division by zero. A class is assigned to the zero-day hold-out set if $S(c) > 1.0$; otherwise, it is retained as a known class.

The 11 known classes form the training and validation basis (benign + 10 attack families), with a 60/20/20 split ($D_{\text{train}}$/$D_{\text{val}}$/$D_{\text{test}}$ by class-stratified sampling).

Data leakage was prevented by fitting all preprocessing (StandardScaler, feature selection) exclusively on $D_{\text{train}}$, calibrating thresholds on $D_{\text{val}}$ only, and isolating $D_{\text{zero}}$ before any preprocessing step; hash-based checks confirmed zero overlap between splits.

Feature selection is performed using a Random Forest classifier (300 trees, max depth 20) fitted exclusively on $D_{\text{train}}$. Gini importance is used rather than permutation importance because of its computational efficiency on large datasets; L2 regularisation ($\lambda = 10^{-4}$) on all dense layers mitigates any residual multicollinearity. A Variance Inflation Factor (VIF) analysis on the 64 selected features confirmed that high-VIF features do not materially degrade zero-day detection rate (Z-DR) when removed ($\Delta$Z-DR $< 3\%$), justifying retention of the full set. We deliberately select exactly $k = 64$ features that naturally reshape into a square grid:

This eliminates need for feature engineering, padding, or augmentation. The 64 features are ranked by importance:

All samples are normalised using StandardScaler fitted exclusively on $D_{\text{train}}$ (fit_transform) and applied via transform to $D_{\text{val}}$, $D_{\text{test}}$ and $D_{\text{zero}}$ to prevent leakage of test-set statistics:

Each 64-dimensional feature vector is deterministically reshaped into an 8 $\times$ 8 2D spatial image:

This reshaping is bijective–no features are added, removed, or synthesized–and enables efficient power-of-2 pooling (8 $\rightarrow$ 4 $\rightarrow$ 2) with standard CNN kernels.

The architecture uses three residual CNN blocks with squeeze-and-excitation (SE) attention for spatial feature extraction, followed by Bidirectional GRU layers for temporal modelling, trained with Binary Focal Cross-Entropy ($\alpha = 0.25$, $\gamma = 2.0$) and AdamW ($lr$ = 5 $\times$ $10^{-4}$) (Figure 1a).

The AE uses a skip-connection encoder-decoder (Dense 1024 $\rightarrow$ 512 $\rightarrow$ 256 $\rightarrow$ 16 $\rightarrow$ 256 $\rightarrow$ 512 $\rightarrow$ 1024 $\rightarrow$ 64) with skip connections from encoder to decoder layers, trained on all known classes with mean squared error (MSE) loss and AdamW ($lr$ = 5 $\times$ $10^{-4}$) (Figure 1b). Reconstruction error (Eq. (7)) serves as the zero-day anomaly signal:

The final detection decision combines both components through a logical OR rule on two independent thresholds (Figure 2b). A network flow is flagged as a zero-day if either the CNN anomaly score $s$ exceeds threshold $\theta_s$ or the AE reconstruction error $e$ exceeds threshold $\theta_a$:

Thresholds $\theta_s$ and $\theta_a$ are calibrated on $D_{\text{val}}$ only, via a percentile grid search (p88–p99) subject to FPR $\le$ 5%, maximising Z-DR on the validation set. $D_{\text{zero}}$ is never accessed during calibration; its role is reporting only. The selected operating point is p97, yielding calibrated values $\theta_s = 0.02104$ and $\theta_a = 0.01104$, chosen as the last stable configuration before a sharp Z-DR cliff at p98. The formal optimisation is:

SHAP (KernelExplainer, 20-centroid background, 30 zero-day flows) was applied independently to both detection components to produce per-feature importance rankings, enabling comparison of CNN and AE signal drivers and explaining complementary detection coverage.

All experiments use the CICIDS2017 dataset [20], which contains 3,119,345 labeled network flows across 15 traffic classes generated in a realistic network environment over five days. After removing the degenerate Portscan class and applying the SLSO split (Section 3.2), the partitions contain: $D_{\text{train}}$ = 1,198,586 flows (11 known classes, 60%); $D_{\text{val}}$ = 399,528 flows (20%); $D_{\text{test}}$ = 399,529 flows (20%); $D_{\text{zero}}$ = 5,546 flows (DoS Slowhttptest: 5,499; Heartbleed: 11; Infiltration: 36). All models are implemented in TensorFlow 2.x and trained on an NVIDIA L4 Graphics Processing Unit (GPU) (24 GB Video Random-Access Memory (VRAM)) with mixed 16-bit Floating Point (FP16) precision and Accelerated Linear Algebra (XLA) Just-In-Time (JIT) compilation. CNN-GRU training uses AdamW ($lr$ = 5 $\times$ $10^{-4}$, weight decay = $10^{-4}$), batch size 512, and early stopping (patience = 10) on validation F1. AE training uses AdamW ($lr$ = 5 $\times$ $10^{-4}$), batch size 512, and early stopping on validation MSE. All preprocessing statistics are fitted exclusively on $D_{\text{train}}$.

Random Forest feature selection (300 trees, Gini importance) on $D_{\text{train}}$ identified 64 features, confirmed non-redundant by VIF analysis ($\Delta$Z-DR $< 3\%$ on high-VIF removal). t-distributed Stochastic Neighbor Embedding (t-SNE) visualization confirms that the three zero-day classes occupy distinct regions relative to known classes (heartbleed: score 74.6, DoS Slowhttptest: 5.09, infiltration: 1.43), validating the SLSO split. StandardScaler fitted on $D_{\text{train}}$ was applied to all sets; hash-based checks confirmed zero sample overlap across all partitions.

The hybrid system was evaluated against three held-out zero-day attack classes: DoS Slowhttptest ($n$ = 5,499 flows), Heartbleed ($n$ = 11 flows), and Infiltration ($n$ = 36 flows). Table 1 reports the overall Z-DR, FPR, Area Under the Receiver Operating Characteristic curve (AUROC), and per-class breakdown for the proposed model alongside four classical anomaly detection baselines.

Baseline hyperparameter settings: Isolation Forest–$n_{\text{estimators}}$ = 200, contamination = 0.05; LOF–$n_{\text{neighbors}}$ = 20, contamination = 0.05, algorithm = ball_tree (novelty = True); Elliptic Envelope–contamination = 0.05, support_fraction = 0.9; Stochastic Gradient Descent-optimized One-Class Support Vector Machine (SGD-OCSVM)–$n_u$ = 0.05, Nystroem radial basis function (RBF) kernel (gamma = scale, $n_{\text{components}}$ = 300), max_iter = 1000. All baselines were fitted on the same training distribution as the AE (all known classes, 1,198,586 flows) and evaluated using the same 95th-percentile threshold on $D_{\text{test}}$ anomaly scores.

The hybrid system achieves 97.48% Z-DR (AUROC 0.956, FPR 4.2%), outperforming all baselines: SGD-OCSVM (85.45%), Elliptic Envelope (60.01%), Isolation Forest (25.46%), and LOF (16.05%). Heartbleed is detected at 100.0%, consistent with its highly distinctive network fingerprint. The Infiltration class presents the most challenging scenario (61.11% Z-DR), attributed to its low anomaly signal (Mean Anomaly Score = 0.014). Bootstrap confidence intervals (CIs) (1,000 iterations, 95% CI) confirm overall robustness [97.1%, 97.9%]; however, the Infiltration class yields a wide CI of [44.4%, 75.0%] (width: 30.6 pp), reflecting its small sample size ($n$ = 36) and representing the primary reliability limitation of this evaluation.

Receiver Operating Characteristic (ROC) curves were computed for three configurations: AE reconstruction error alone (AUROC = 0.9221), CNN anomaly score alone (AUROC = 0.9556), and the hybrid composite using OR-fusion (AUROC = 0.9560). The hybrid strictly dominates both single-signal configurations. Critically, AUROC does not capture per-class complementarity: the AE alone detects Heartbleed at 100% while CNN alone achieves only 9.09%–a 90.9 pp gap invisible in aggregate ROC curves. The composite score $S = $ max-normalised OR-fusion of CNN anomaly score and AE reconstruction error was used for AUROC computation, consistent with the hybrid decision rule.

Table 2 reports ablation results across three configurations under identical thresholds ($\theta_s$, $\theta_a$).

The CNN-GRU Binary alone achieves 79.23% overall Z-DR at the lowest FPR (2.74%), but it critically fails to detect Heartbleed (9.09%), indicating limited generalisation to this attack’s feature profile. In contrast, the AE v2 alone detects Heartbleed at 100% but performs poorly on DoS Slowhttptest (44.74%), reflecting its sensitivity to structural outliers rather than behavioural patterns captured by the CNN. The full hybrid model (Figure 3), using OR fusion of the CNN anomaly score and AE reconstruction error, achieves the highest overall Z-DR of 97.48%, representing an 18.25 percentage-point improvement over the CNN-GRU Binary alone and a 52.56 percentage-point improvement over the AE v2 alone, with only a modest FPR increase from 2.74% to 4.20%.

Detection pathway analysis on $D_{\text{zero}}$ confirms mechanistic complementarity: 2,915 flows were detected exclusively by the CNN, 1,012 exclusively by the AE (including all Heartbleed flows missed by the CNN), and 1,479 by both components; 140 flows (2.52%) were missed by both. This breakdown confirms that OR-fusion captures 1,012 attacks that the CNN alone would miss entirely.

Threshold calibration across percentiles p88–p99 (Table 3) reveals three operating regions: a stability plateau (p88–p94) where Z-DR holds at 98.14% across a wide FPR range (12.71%–7.27%); a gradual trade-off zone (p94–p97) yielding a 3.07 pp FPR reduction at only 0.66 pp Z-DR cost; and a sharp cliff at p97–p98 where Z-DR collapses by 52.51 pp for a mere 0.99 pp FPR gain–an extremely unfavourable trade-off. Heartbleed remains at 100% throughout; Infiltration degrades steadily from 88.9% to 61.11%, reflecting its low anomaly signal. The selected p97 point ($\theta_s$ = 0.02104, $\theta_a$ = 0.01104) is the last stable configuration before the cliff.

Real-time deployability requires sub-millisecond per-flow inference latency. Table 4 reports GPU inference throughput for each component. All measurements were taken on an NVIDIA L4 GPU (24 GB VRAM) with mixed FP16 precision and XLA JIT compilation. The hybrid sequential pipeline achieves 14,201 samples/second (0.070 ms/sample), well within real-time requirements for typical enterprise traffic (10K–50K flows/minute). At 1 Gbps line rate (approx. 150K flows/minute), approximately 11 parallel GPU instances would be required; Central Processing Unit (CPU)-only inference is estimated at 3–5 $\times$ slower.

To interpret the detection decisions of both components, SHAP values were computed for 30 held-out zero-day flows using KernelExplainer. The CNN anomaly score and AE reconstruction error were each treated as scalar black-box functions of the 64 scaled input features, with a 20-centroid k-means background summary derived from 100 known training flows. Figure 4 shows the top-15 most influential features for each signal.

SHAP analysis (Figure 4) shows CNN detections are driven by temporal features (Active Min, Flow Duration, flow inter-arrival time (IAT) Max), while AE detections are driven by backward IAT statistics (backward IAT Mean, backward Packets/s) absent from the CNN top-15–confirming the mechanistic basis for complementary coverage between the two components.

This paper presented a hybrid CNN-GRU-Attention and skip-connection AE framework for zero-day attack detection, evaluated on CICIDS2017 under strict data isolation. The system achieves 97.48% zero-day detection rate at 4.2% FPR and AUROC of 0.956, substantially outperforming all single-component and classical baselines (CNN-alone: 79.23%; AE-alone: 44.92%; best baseline SGD-OCSVM: 85.45%). SHAP analysis confirms mechanistic complementarity: CNN detections are driven by temporal flow features, while the AE contributes 1,012 exclusive detections via backward IAT statistics invisible to the supervised component. The 8 $\times$ 8 spatial reshaping of 64 selected features enables efficient CNN processing without arbitrary feature engineering, and the system operates at 14,201 samples/second on GPU, supporting real-time enterprise deployment.

The primary limitation is the Infiltration class detection rate (61.11%, CI [44.4%, 75.0%]), reflecting its small evaluation sample ($n$ = 36) and low anomaly signal. The 4.2% FPR, while operationally acceptable given zero-day prevalence, warrants cost-sensitive threshold tuning for high-volume SOC environments. Generalizability beyond CICIDS2017 and robustness to adversarial evasion remain open questions.

Future work will validate the framework on CICIoT2023 (with adapted feature selection to accommodate different feature spaces), investigate adversarial robustness, and deploy in live SOC environments with telemetry feedback for continuous threshold adaptation.