Minimizing Chaos in Echo State Networks: A Hybrid Approach Using the Lorenz System

Deep learning, a subfield of machine learning, has revolutionized various scientific domains by enabling accurate predictions in complex systems [1]. One particular area that poses persistent challenges is the accurate prediction of chaotic systems. Chaos is the behavior of a system that is sensitive to initial conditions [2]. If a system is nonlinear and chaotic, it is impossible to predict its future in most cases [3].

One classic example of a chaotic system is the Lorenz system, which was developed by meteorologist Edward Lorenz in the 1960s. The Lorenz system is a set of three coupled, nonlinear differential equations that describe the behavior of a simplified model of atmospheric convection with sensitive initial conditions. It illustrates the concept of chaos and the 'butterfly effect’ [4]. This latter, hampers reliable forecasts and a comprehensive understanding of their dynamics. Prediction of chaotic systems is needed in today’s world. So we need to minimize the butterfly effect in these systems to get accurate future predictions.

To tackle the butterfly effect and enhance prediction accuracy, researchers have explored innovative approaches in recent years, with a particular focus on neural networks [5-8]. ESN is a recurrent neural network with a unique reservoir computing paradigm. It is characterized by nonlinear calculations, has echo state characteristics, and exhibits the characteristics of forgetting input [9-32]. These characteristics make the ESN a compelling candidate for minimizing the butterfly effect in chaos forecasting. Recent studies have provided empirical evidence for the efficacy of the ESN in chaotic time series prediction.

A lot of ESN-based models [9-21] are developed in this regard. Every new model was getting better results, but they were still lacking to map slow behavior chaotic systems as artificial neurons [1]. This study is providing a detailed literature survey on these models in the next section.

The Lorenz system [22] exhibits chaotic behavior with initially similar trajectories that diverge over time. By leveraging the insights from the Lorenz system and combining them with the power of the ESN, our research aims to develop a hybrid approach that further minimizes the butterfly effect and enhances prediction accuracy.

This paper introduces a novel hybrid approach that integrates the Lorenz system and the ESN. The Lorenz system provides a foundation for understanding chaotic behavior, while the ESN leverages its deep learning capabilities to model and predict complex dynamics. By utilizing training data from the Lorenz system, our hybrid approach trains the ESN to assimilate the divergent trajectories and capture the underlying dynamics. The synergistic fusion of the Lorenz system and the ESN offers a promising solution to address the challenges posed by chaotic systems, leveraging the power of neural networks and deep learning techniques.

This research got remarkable results with a minimum RMSE of 1.0. This hybrid approach is outperforming the original Lorenz model in every case. This model can capture complex dynamics in various domains and it is very advantageous in stock forecasting.

The remainder of this paper is structured as follows: Section 2 provides a comprehensive review of the Lorenz system and the ESN, highlighting their respective contributions to chaos forecasting. Section 3 contains the methodology behind our hybrid approach, including the data collection, training process, and prediction framework. In Section 4, we present the simulation setup and discuss the results, demonstrating the efficacy of our approach in minimizing the butterfly effect. Section 5 concludes the paper by summarizing the key contributions and underscoring the significance of our hybrid approach in advancing chaos forecasting techniques.

Chaos is a nonlinear dynamical system's behavior that is incredibly sensitive to even little changes in the original circumstances [2]. Future projections of a chaotic system can drastically shift if the beginning conditions are even slightly altered [23]. Because of this property, chaotic systems can become difficult to forecast accurately [23]. These systems have no periodic behavior such as oil and gas systems, weather forecasting systems, financial market systems, hydrological systems, etc. [2].

The butterfly effect is the sensitive dependency of chaotic to initial conditions. This phrase was one of the most famous phrases from 20th-century science [4]. This term was originally generated by Ed Lorenz in his paper in 1963 [22]. But it was coined by Gleick in his famous book written on chaos [24].

For the prediction of chaotic systems, a lot of models are proposed. First of all, machine learning was used [1]. Some algorithms that were applied are artificial neural network (ANN), feedforward neural networks [6, 7, 33], support vector machine (SVM) [5], LSTM with recurrent neural networks (RNNLSTM) [8], and reservoir computing (RC) [25-28]. From these techniques, RC became the most famous technique because of its better predictions than other models [1]. Then, the prediction time of reservoir computing increased by using it as a hybrid model [29]. After that, the challenge was to decrease computing costs and grow the prediction horizon. Then, researchers proposed ESN [30]. This algorithm was used with a lot of variations and it gave remarkable results.

The echo state network was used with different variations. In this paper, those models are reviewed to show the effectiveness of this study. Table 1 contains those ESN models with their authors and publication years.

The ESN proposed by the study [9] consists of an input layer, a hidden layer (reservoir having nodes), and an output layer [10]. In 2006, a model named as sigmoid-wavelet hybrid ESN (SWHESN) [11] was developed to upgrade the performance of ESN. It increased the memory capacity of ESN. It reserved the nonlinear feature of ESN by inserting tuned wavelet neurons into it. It provides 46% more prediction accuracy than ESN. all achieved in just 30% of the time it took for ESN. Continuing the evolution of reservoir networks, LIESN was introduced in [12]. LIESN introduced a novel algorithm with global control parameters.

This model was able to categorize noisy and slow time series. In [13], a model named 𝜑-ESN was proposed with four main factors including various time scale dynamics, input variability, regression in argument attribute space, and nonlinear relation in units. Building on this progress, the R2SP model [14] added static layers to the dynamic reservoir system for improving accuracy.

Meanwhile, MrESN, as described in [15], took a different approach by utilizing multiple reservoirs for the projection of multivariate chaotic time series. In this model, one multivariate time series was related to one reservoir. This model was getting a better accuracy with a root mean square error of 43.70.

A novel model was introduced in [16] known as squared root cubature Kalman filter-γ echo state network (SCKF-γESN). In this model, γESN was used for the modelling of multivariate time series and then SCKF was used to upgrade parameters of it. For the security of the model, it was protected by using an outlier detection algorithm. This model was used online for later forecasting.

The pursuit of hierarchical structures within reservoir networks led to the development of ML-ESM [17] and Deep ESN [18]. These models aimed to add depth and hierarchy to the reservoir network, facilitating the learning of complex multiscale dynamics.

DeePr-ESN [19] took this concept even further, demonstrating its ability to capture intricate multiscale dynamics effectively. It became a powerful tool for handling diverse time series data.

In the study [10], a model names SOGWOESN was proposed which was improved by Grey Wolf optimizer (GWO). This model got a maximum optimizing ability percentage of 78%.

In the study [20], double reserve pools were used with ESN for power load prediction. The model was trained with historical data, environmental data, and ESN parameters with double reserve pools. BFA-DRESN algorithm improved the forecasting accuracy with the RMSE value of 18.83. Finally, HDESN [21] was proposed for multistep chaotic time series prediction. It was able to get expansion patterns through hierarchical processing and deep topology. And it got satisfactory performance in chaotic forecasting.

In all of the above cases, ESN indeed demonstrated its potential for continuous improvement and adaptability in various applications, consistently yielding better results with each new research endeavor. However, despite its remarkable progress, it is essential to acknowledge that ESN still faces certain limitations. Notably, a significant challenge lies in its inability allowing for the seamless integration of chaotic systems with slow-behavior as artificial neurons, as discussed in [1]. This constraint underscores a crucial area for further exploration and innovation, as enhancing ESN's capacity to handle such complex systems could unlock even more transformative possibilities in the realm of reservoir networks.

One well-known example of a chaotic system is the Lorenz attractor. It is named after the mathematician Edward Lorenz, who studied it extensively in the 1960s [22-31]. As part of his research on the predictability of weather patterns, Lorenz found that even minor changes to the system's initial conditions could have a significant impact on the system's long-term behavior. The butterfly effect, or the sensitivity to intial conditions, is a characteristic of chaotic systems. Lorenz’s discovery of the Lorenz attractor led to important insights into the limits of predictability in complex systems, and it has had a major impact on the study of nonlinear dynamics and chaos theory. The Lorenz attractor has also been used as a model for a wide range of physical and biological systems, and it continues to be an important subject of study in mathematics, physics, and engineering.

Bitcoin stock prices data were collected from the Kaggle website, the data have three parameters (price, high and low) expressed by (x, y, z) that we used to construct the Lorenz system, which is characterized by the following equations Eq. (1) [14].

By solving the equations of the Lorenz system, specifically Eq. (1) with the given parameters, we witness a mesmerizing phenomenon: the solution forms a remarkable butterfly shape when plotted in three-dimensional space, as beautifully depicted in Figure 1. It is from this distinctive shape that the renowned concept of "The butterfly effect" derived its name. The butterfly shape observed in the solutions of the Lorenz system serves as a captivating representation of its chaotic attractor. As we trace the trajectories of the state variables (x, y, and z), they intertwine and fold upon themselves, creating an enchanting pattern reminiscent of a butterfly in flight.