A Comprehensive Evaluation Index for GNSS Observation Data Quality

GNSS refers to all the satellite navigation systems. With over 130 navigation satellites [1], GNSS can provide users with three-dimensional coordinates and time information on the Earth's surface or near-Earth space, achieving functions such as positioning, velocity measurement, and timing. Leveraging its advantages of all-weather operation, high precision, and no line-of-sight requirements, it can conduct periodic or real-time monitoring of landslides, making it widely used in geodesy, navigation and positioning, and geological surveying [2], [3].

The accuracy of GNSS positioning results is closely related to the quality of observation data, which is mainly affected by the surrounding observation environment and the performance of the receiver itself [4], [5], [6]. In practical monitoring applications, landslides often occur in outdoor environments, causing the receiver to be easily obstructed by surrounding trees, mountains, and other environmental factors when receiving satellite signals in the field, which affects the quality of observation data and reduces the accuracy and reliability of landslide positioning and warning.

However, the traditional evaluation indicators such as signal-to-noise ratio (SNR), cycle slip ratio (CSR), multi-path error effect and other indicators can only reflect the specific data quality, and are not suitable for evaluating the overall quality of observations. The single entropy weight method only starts from mathematical statistical characteristics, ignores the actual significance of indicators, and cannot reflect the actual importance of indicators. It is extremely sensitive to extreme values in sample data. If there are extreme values in the data, it will cause a drastic change in the entropy value of a certain indicator, thereby distorting weight allocation. Moreover, its weight is completely determined by the degree of data dispersion. If the data fluctuation of a certain indicator is small, its entropy value may be underestimated, resulting in unreasonable weight allocation.

Overall, in the face of such complex GNSS observation data, it is difficult for traditional quality analysis indicators and single weighting methods to comprehensively and accurately characterize the overall quality of the multi-system observation data. Therefore, on the basis of these two methods, the weight coefficient of each evaluation index was determined by combining the entropy weight method and the coefficient of variation method [7], and then combined with the fuzzy mathematical theory to obtain the quality evaluation results in this study. The results show that the proposed method can accurately reflect the poor-quality satellite system data in the observation data, and provide a reliable reference for the subsequent positioning solution.

The quality of observed data is a critical guarantee for the accuracy and reliability of satellite navigation and positioning. It is influenced by various factors such as the observation environment, the hardware performance of the receiver, and the status of the satellite.

Currently, the International Organization for Standardization (ISO) has long recognized multi-path effect, CSR, SNR, data efficiency, and Geometric Dilution of Precision (GDOP) is the core evaluation index of GNSS data quality [8]. The above indicators are independent of each other, covering the four key dimensions of the equipment's working environment, signal, data continuity and satellite geometry. They can fully cover the key error sources of the whole link of satellite signals from launch, propagation, reception and calculation. According to the empirical value, data statistics and relevant literature [9], [10], the influence degree of each index on the quality of observation data can be divided into four levels: Tier I considers the observation data to be ideal with high confidence, suitable for direct use in positioning; Tier II considers the observation data to be good, with some epochs having poor quality; Tier III considers the observation quality to be poor, which will significantly affect positioning accuracy; and Tier IV considers the quality to be very poor, unsuitable for use in positioning solutions, and should be discarded, as shown in Table 1.

Data efficiency is a crucial metric for assessing data quality. It is defined as the ratio of actual valid data received over a period to theoretical observed data. Theoretical data is calculated based on the set satellite elevation angle and satellite ephemeris, which determines the number of satellite observations that can be received. However, during actual observation, due to environmental factors around the receiver and hardware limitations of the receiver itself, some unusable satellite observation values are removed. The actual number of received data after such removals constitutes the valid data. A higher ratio indicates better data efficiency and greater completeness and reliability. Its calculation formula is given as follows:

where, $Ra$ represents the effective rate, $Hav$ represents the actual number of receptions, and $Exp$ represents the theoretical number of receptions.

The foundation of satellite positioning is based on calculating the distance between a satellite and a receiver by measuring the time it takes for a signal to travel from the satellite to the receiver. However, during actual signal propagation, environmental factors such as reflection and refraction can affect the signal, causing multi-path signals to have longer propagation paths than direct signals. This results in significant positioning errors [11], as shown in Figure 1. Nowadays, linear combinations of pseudorange and carrier phase observations can be used to estimate multi-path effects for dual-frequency observations of all the GNSS [12], with the formula given as follows:

where, $M P$ is the multi-path effect; $P$ is the dual-frequency pseudorange observation value; $L$ is the dual-frequency carrier phase observation value; $f$ is the frequency; and $k, i$ and $j$ are the frequency point numbers.

The $M P$ value reflects the change of the multi-path effect. The smaller the $M P$ value is, the stronger the ability to resist the multi-path effect is, and the less influence it has on the final positioning solution.

Cycle slip is the phenomenon of discontinuity in the carrier phase caused by various factors when a receiver receives satellite signals. Factors that can cause cycle slip include brief occlusions by surrounding objects, rapid movement of the receiver, and hardware failures of the receiver.

CSR, as an indicator of error measurement in GNSS, represents the ratio between the overall observation and the number of times jitter occurs during the signal reception process. A higher value is better. Over the past few decades, many algorithms have been proposed for detecting and correcting jitter ratios, including phase ionospheric residuals [13], Kalman filter-based methods [14], polynomial fitting [15], [16], and wavelet transform [17], [18].

Anubis utilizes the combination of geometry-free and Melbourne-Wubbean methods to continuously detect weekly slips in the study by Leick et al. [9]. By calculating the phase change of each satellite at each epoch and using differential algorithms to calculate the phase changes between adjacent epochs, it determines whether a weekly slip has occurred. Existing scholars have used the CSR method to evaluate the weekly slip ratio, with the calculation formula shown in Eq. (3). The smaller the value, the fewer the number of weekly slips during the monitoring period, indicating better quality. In the formula, $O/Slip$ represents the weekly slip ratio.

SNR refers to the ratio of the strength of a satellite signal to the intensity of environmental noise, measured in dB·Hz. It is primarily influenced by factors such as the satellite's elevation angle, receiver antenna, and the multi-path error. The higher the SNR, the stronger the signal and the better the quality of the observed signal. Furthermore, studies have shown that the SNR can also indirectly reflect the environment around the receiver, allowing for the reduction of multi-path effects through appropriate algorithms [19]. Its calculation formula is given as follows:

where, $S N R$ represents the signal-to-noise ratio, $P_s$ represents the signal power, and $P_n$ represents the noise power.

In the field of satellite positioning, the precision factor is an indicator that measures the geometric conditions possessed during GNSS measurements, and its value is related to the distribution of receivers and satellites. GDOP reflects the amplification effect of the geometric distribution between the station and satellites on ranging errors. When the angles between the receiver and satellites are close and all observable satellites are concentrated in a certain area, the GDOP value is high, leading to poorer positioning accuracy [20]. In actual observations, it is necessary to ensure that satellites are evenly distributed across different directions.

At the end of the $19{ }^{\text {th }}$ century, the famous German mathematician Cantor first proposed set theory, which greatly promoted the development of mathematics. In 1956, American Chad proposed the concept of fuzzy set on the basis of set theory [21], which is mainly used to describe the uncertainty and randomness of the objective world, and has been widely used in various fields. In the traditional set theory, it is believed that for an element $u$ of a set $A$, there are only two states: either it belongs to the set $A$ or not. There will be no third situation.

However, in real life, many situations are not “either-or.” Instead, they involve more ambiguous concepts where it is impossible to give an absolute yes or no answer to whether each object fits. The answer lies between “good” and “bad” and “yes” and "no," presenting a relationship that is both this and that. Therefore, it is necessary to extend the results of characteristic functions from traditional set theory, which can only take values of 0 and 1, to the closed interval of [0,1]. This means that different elements in a set can have different degrees of membership within the same set, thereby quantifying the relationship between elements and sets.

To reflect the mapping relationship between fuzzy subsets and membership functions, it is necessary to establish a membership function. For a given fuzziness set $A$, the membership function $u(x)$ can map element $x$ to $[ 0,1]$. Common membership functions include triangular membership functions, trapezoidal membership functions, and Gaussian membership functions, among others. In depicting the grading boundaries of GNSS data quality, a reduced half-trapezoidal distribution was used to characterize the membership. Finally, these three membership functions were brought into Table 1 to obtain the specific membership functions.

In the trapezoidal membership function, the partial membership function is as follows:

The intermediate membership function is as follows:

The defuzzification membership function is as follows:

The comprehensive evaluation index proposed in this study is based on the following fundamental idea: using two weighting methods—entropy weight method and coefficient of variation method—to obtain the weight coefficients for each satellite system in the observed data through linear combination. Here, the “entropy” in the entropy weight method refers to information entropy, which reflects the degree of disorder in information and can measure the amount of information. In the quality assessment of observed values, the more information an indicator carries, the smaller its entropy value, indicating that the indicator plays a greater role in quality evaluation.

However, the weight obtained by the entropy weight method is derived from actual observations, which, while reducing the impact of outliers, also has the drawback of weight distribution equilibrium [22]. This is because the entropy weight method determines weights based on observed data without considering the varying degrees of influence that different evaluation indicators have on the quality of observed data. Therefore, the weights should not be balanced. The weights from the entropy weight method were adjusted using the coefficient of variation method, and then the comprehensive evaluation index for each satellite system in the observed data was derived through linear combination. A higher value indicates better quality.

Standardized data processing: Since the dimensions of various indicators are not the same, the data need to be normalized. According to the evaluation relationship between indicators and quality, it can be divided into the following two formulas (forward pointer and negative indicators):

where, $Q_{i j}$ is the $j$-th sample data under the value of the $i$-th index after dimensionalization, $X_i$ is the $j$-th sample data of the $i$-th index, and ${\operatorname{Min}}\left(X_i\right)$ and ${\operatorname{Max}}\left(X_i\right)$ represent the minimum and maximum values of the sample under the $i$-th index, respectively.

The proportion of the j-th sample under the i-th index can be calculated as follows:

The entropy value and the weight of index $i$ can be calculated as follows:

where, $e_i$ represents the entropy value of index $i$, and $D_i$ represents the weight of index $i$.

Coefficient of variation in the legal authority process is as follows:

where, $C V$ is the coefficient of variation, $P$ is the weight based on the coefficient of variation, and $\sigma$ and $\mu$ are the standard and mean values, respectively.

The weighting factor in the combination weighting can be calculated according to the variance contribution rate as follows:

The final combination weight was constructed by combining the two results through the weighting factor as follows:

To verify the effectiveness of the method presented in this study, test validation was conducted using actual measurement data. The test data was obtained from the monitored and measured data of the same slope from 00:00 to 12:00 on different dates (August 13, 2024, September 29, 2024 and October 20, 2024). The calculation process takes the station data on August 13 as an example. Data corresponding to the observation values from the station were extracted and used to calculate the effective availability, the multi-path effect index, SNR, and CSR, among others. The results are shown in Table 2.

The observed data index was brought into the corresponding membership function, and the fuzzy degree relationship matrix of each satellite system capacity was obtained as follows:

After normalizing the data, the information entropy weighting method was used to calculate the information entropy to obtain the weight value. Therefore, the observed data was substituted into the above formula to obtain the weight coefficient as follows:

$B=w \cdot R$ was used to obtain the fuzzy comprehensive evaluation results of GNSS observation data quality based on the information entropy weighting method. The results are shown in Table 3.

The evaluation results can also be roughly divided into four levels: Level I considers the observation data to be ideal with high confidence, suitable for direct use in positioning; Level II considers the observation data to be good, but some historical data have poor quality; Level III considers the observation quality to be poor, which will significantly affect the positioning accuracy; and Level IV considers the quality to be very poor, unsuitable for use in positioning calculations, and should be excluded.

Firstly, the characteristic value matrix of the evaluation index was constructed according to the data in Table 1 as follows:

$\left[\begin{array}{ccccc}50 & 5 & 20 & 95 & 1 \\ 40 & 10 & 40 & 90 & 6 \\ 35 & 15 & 60 & 80 & 8 \\ 30 & 20 & 80 & 70 & 20\end{array}\right]$

The weight based on the coefficient of variation method was calculated by Eqs. (10)-(11), and finally the weight coefficient of the combined weighting after the weight adjustment based on the coefficient of variation method was obtained by Eqs. (12)-(13),

The fuzzy comprehensive evaluation results of GNSS observation data based on combination weighting were obtained by using $B=w \cdot R$, as shown in Table 4.

The quality analysis index in the observation data, the comprehensive evaluation model based on the information entropy weighting method and related results are summarized in Table 5.

As shown in the table above, using a single evaluation metric for assessing GNSS observation quality has obvious limitations, leading to contradictory results from different single metrics. For example, under the multi-path metric, GLO quality is rated as Level III, indicating poor quality and low confidence; however, under the data efficiency metric, GLO quality reaches its best level. The inherent flaws of this single-metric evaluation system prevent it from comprehensively reflecting the overall system quality, making it difficult to balance the weight relationships among various quality metrics and lacking a unified standard for assessment.

The information entropy weighting method calculates the weights of each indicator and then combines them through linear methods to encompass all aspects of observational data. It can represent the overall quality of observations through its composite indicators, indicating that the evaluation results obtained using the information entropy weighting method are more reliable and reasonable compared to those from single indicators. The calculation results of the information entropy weighting method show that the observation values of GPS and BDS have excellent quality and high confidence, making them suitable for direct use in positioning solutions. The observation data of GAL and GLO also have good quality, with most epoch data being usable for positioning solutions. According to the calculation results of the information entropy weighting method, the data from certain epochs of GLO and GAL should be excluded in data processing.

The comprehensive evaluation model constructed using a combination of information entropy weighting and coefficient of variation shows that the quality of BDS and GPS satellite observation data is equally excellent with high confidence levels, meeting the quality requirements for high-precision positioning calculations, and can be directly used in subsequent positioning calculations. However, the quality of GLO satellite observations is poor, requiring further assessment of application risks. The GAL observation data are the most severely affected with very low confidence levels, and should be excluded in subsequent positioning calculations to avoid impacting overall positioning accuracy.

The results obtained from the combined weighting method differ slightly from those of the information entropy weighting method. This is because the combined weighting method takes into account not only the actual observed conditions but also the overall impact of indicators on observation quality. By adjusting the weights derived from the information entropy weighting method using the coefficient of variation, its weight coefficients tend to be more reasonable compared to those obtained solely through the information entropy weighting method.

The weights obtained from the information entropy weighting method and the combined weighting method are summarized in Table 6. According to the table, under the information entropy weighting system, the weight coefficients of each indicator are ranked as follows: data completeness $>$ SNR $>$ geometric accuracy factor $>$ multi-path effect $>$ CSR. Under the combined weighting method, the weight coefficients of each indicator are ranked as follows: geometric accuracy factor $>$ multi-path effect $>$ CSR $>$ SNR $>$ data completeness. In the combined weighting method, the geometric accuracy factor has the highest weight coefficient, indicating that this indicator is the most important. This is because the geometric accuracy factor encompasses both horizontal and vertical satellite position distributions. Additionally, when the satellite's elevation angle is too low, it can lead to longer signal propagation paths, which also increases the multi-path effect value. Since this indicator covers the most information, its weight should be the largest. The multi-path effect directly impacts positioning accuracy and should not have an excessively small weight. Conversely, since the SNR and data completeness of each satellite system are relatively high and not significantly different, they should not be considered the most influential factors. This indirectly proves that compared to the information entropy weighting method, the combined weighting method is more scientifically accurate.

According to the above calculation process, the calculation results of the other two days were calculated respectively, as shown in Table 7.

The solutions can be categorized into fixed and floating solutions based on whether the ambiguity is fixed. A floating solution is obtained when the ambiguity cannot be fixed, while the accuracy of a fixed solution is much higher than that of a floating solution. The success rate of fixing the ambiguity directly reflects the quality of the solution. In the solution result file, $Q=1$ indicates a fixed solution, and $Q=$ 2 indicates a floating solution. The solution mode shall be consistent, and the specific settings are shown in Table 8. Due to the selected same slope, it is considered that environmental factors have basically the same impact on the observation data. From the above analysis, it can be concluded that during monitoring, the combined index values of GLO and GAL are relatively low, significantly interfering with the positioning results. To verify whether the comprehensive index can accurately reflect the data quality of the observation value, the observation value was solved using different positioning methods, and the quality of the positioning results was reflected through the success rate of fixing the ambiguity. In the final positioning solution, the satellite systems involved in the positioning were divided into two groups: GPS + BDS + GAL + GLO, and GPS + BDS. The positioning results are shown in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6.

The above calculation results were summarized, obtaining the information in Table 9.

The mean value of the standard deviation in the above table represents the average standard deviation of the baseline length during the monitoring process, indicating the stability of the baseline during the whole monitoring process. Therefore, it can be seen from the table that the mean value of the standard deviation of the multi-system combination (GPS + GLO + GAL + BDS) is significantly higher than that of the GPS + BDS combination. At the same time, the success rate of ambiguity fixation of the two systems is significantly higher than that of the other system combinations, indicating that the higher the success rate of ambiguity fixation, the more stable the baseline solution results. This indicates that the data quality of GAL and GLO is poor, which has a great impact on the positioning results. The experimental results are consistent with the comprehensive indicators, and it is also verified that eliminating the satellite system data with poor quality has a significant effect on improving the positioning accuracy and stability.

A GNSS observation quality assessment method based on fuzzy mathematics theory was proposed in this study. Five evaluation indicators were selected, namely, average SNR, average multi-path effect, data efficiency, CSR, and the average DGOP value. After combining the entropy weighting method with the coefficient of variation method for weighting, these indicators were used in conjunction with fuzzy mathematics theory to characterize the quality of observation data. Experimental data show that this indicator can accurately and scientifically reflect the quality of observation data, making it an effective evaluation metric for GNSS observation data, suitable for application in engineering experiments. However, the number of samples in this experiment is small. Future experiments could increase the sample size to further verify the reliability of this indicator.