Machine Learning-Based Road Level-of-Service Classification Using K-Nearest Neighbours and Shapley Additive Explanations Analysis

As is well known, road safety has a critical impact on human life, influencing economic, environmental, social, and political outcomes. A review of the literature reveals that determining road quality is essential for ensuring traffic safety and developing intelligent traffic systems. In traffic engineering, this concept is commonly represented by the level of service (LOS) [1,] [2]. With recent advancements in technological tools, traffic management has become a fundamental component of smart city infrastructure, and traffic prediction plays a key role in real-world applications.

Historically, studies on LOS were first formalized in the Highway Capacity Manual (HCM). Six levels, from A to F, were established to classify varying traffic conditions. In the 1965 edition of the HCM, travel time and the flow-to-capacity ratio were identified as critical factors for LOS. In the 1985 edition, traffic flow, travel speed, and traffic density were highlighted. The 2000 edition further emphasized travel time and speed, as well as driver maneuverability, comfort, and convenience, and interruptions experienced while traveling [3].

With the advent of modern technologies, traffic-related challenges have found new potential solutions. As a result, LOS prediction has increasingly been explored through computational methods, including fuzzy logic and artificial neural networks (ANNs) [4], [5], [6], [7], [8]. For instance, Pattnaik and Ramesh Kumarb [9] evaluated speed, volume, and acceleration noise within the HCM framework using various fuzzy sets. Basu et al. [10] employed a neural network approach to assess vehicle equivalency, using stream speed as a performance measure and analyzing the effects of traffic volume and composition.

In recent years, newer analytical approaches such as Shapley Additive Explanations (SHAP) and machine learning (ML) have been increasingly adopted [11], [12], [13], [14]. SHAP has been widely used in traffic safety studies [15], [16], [17]. However, its application to LOS prediction remains limited, making it a promising and valuable direction for future research. For instance, Biswal and Bhuyan [18] applied SHAP to predict LOS at intersections, identifying key features that influence LOS at signalized intersections. Additionally, ML methods have proven effective in exploring LOS and traffic congestion. For example, Park et al. [19] and Yin et al. [20], investigated road types and congestion levels using various ML techniques.

Therefore, it is evident that the integration of these advanced methods and a deeper understanding of parameter relationships can yield valuable insights for traffic management and LOS prediction.

This study aimed to accurately predict the LOS using ML techniques. Four different algorithms, K-nearest neighbours (KNN), support vector classification (SVC), random forest (RF), and AdaBoost, were applied for this purpose. Before model development, a correlation matrix was utilized to examine the relationships between input features and LOS classes. The models were subsequently evaluated using various performance metrics, including accuracy, Cohen’s Kappa, receiver operating characteristic (ROC) curves, and Area Under the Curve (AUC) scores, providing a comprehensive comparison of their classification capabilities.

The paper is organized as follows: Section 2 provides an overview of the methodology and describes the dataset used in the study. It also details the methods applied and the evaluation metrics employed; Section 3 presents the results obtained; Section 4 provides the conclusions of the study.

In this study, a structured approach was followed to generate ML models aimed at predicting LOS. Initially, a correlation matrix was generated to investigate the relationships among variables. Based on these insights, four classification algorithms, KNN, SVC, RF, and AdaBoost, were selected and implemented. Model performance was evaluated using standard evaluation metrics such as accuracy, Cohen’s Kappa, ROC curves, and AUC values.

This study used traffic counts on the vehicle access road in Hacettepe University Sıhhiye campus hospital, a critical area for vehicle traffic [21]. The counts were taken at 15-minute intervals during peak hours with high entry and exit traffic. The statistical information of the parameters used as input in the ML models is given in Table 1, and the Box and whisker plot is shown in Figure 1.

In this study, the LOS classes (A–E) were determined based on the threshold criteria specified in the HCM for uninterrupted-flow facilities. For each observation, key performance indicators such as average speed and volume-to-capacity (V/C) ratio were calculated and compared with the corresponding HCM-defined ranges. Observations with higher average speeds and lower V/C ratios were categorized as LOS A or B, representing free-flow conditions, whereas those with lower speeds and higher V/C ratios were classified as LOS D or E, indicating unstable or near-capacity flow.

As is known, ML is a set of methods that utilize many algorithms. ML-based algorithms offer different analysis techniques for an adopted algorithm. ML methods typically encompass three primary learning techniques: reinforcement, supervised, and unsupervised [22]. ML has recently been experiencing its heyday as it has a popular usage area with the possibility of being used by users in scientific studies and branches covering many different subjects in the field of industry. In recent studies with the ML technique, it has come to the forefront with different usage areas, such as its use in the field of medicine [23]. Its use in heart attack prediction [24], Its use in agriculture [25], its use in precipitation prediction [26], Its use in predicting the optimum binder content of asphalt pavement [27], Its use in intelligent transportation systems [28].

In this study, KNN, SVC, AdaBoost, and RF algorithms were selected. The selection of these algorithms was based on their representation of different learning paradigms and their adaptability to various types of data in both classification and regression problems. In this study, four different ML methods were used to predict five different service levels in the road network of the specified area, with A being the most accurate and E being the least accurate. The ML methods used in the study are given in Table 2.

KNN is a commonly applied ML algorithm positioned under supervised learning techniques for both classification and regression tasks [29]. Due to its simplicity and strong generalization performance, the KNN classification method is recognized as a commonly applied approach. The KNN algorithm classifies unlabeled instances by classifying them into the category of the closest labeled examples [30]. Each data entry in the dataset is associated with a corresponding label, and the KNN algorithm determines the class of new instances based on these related labels [31].

The choice of an appropriate $k$ value is critical in classification tasks. Choosing the right $k$ value for a given dataset is often challenging due to the limited size of the training set. While many KNN approaches operate using a single predefined $k$ value, employing different $k$ values based on data boundaries can significantly improve classification accuracy [32]. For variable samples ($x, xi$), the Euclidean distance used in KNN is defined by a specific equation [33]. Higher accuracy is typically achieved with low-dimensional datasets. In contrast, when the dataset is high-dimensional and sparse, the Manhattan distance is used, as formulated in the corresponding equation [34].

Here, $x$ and $y$ represent vector variables.

Certain criteria influence the performance success of the KNN algorithm. These include the choice of $k$, the distribution of the data, the distance metric used, as well as the algorithm’s susceptibility to noise and outliers [35].

SVC is a robust and widely recognized type of supervised ML algorithm that has been employed for classification task across different fields [36]. In recent years, the method has garnered increasing interest from researchers due to its ability to deliver high generalization performance, making it a preferred choice for accurately distinguishing between different data classes in complex classification tasks [37], [38]. The principle of SVC lies in identifying the most appropriate hyperplane that effectively constructs a decision boundary between different classes, thereby maximizing the margin between them and leading to improved generalization effectiveness [39].

AdaBoost is a widely utilized algorithm within the ensemble learning paradigm of ML. Its fundamental purpose is to enhance classification accuracy by iteratively combining multiple weak classifiers into a single, strong learner [40]. The algorithm initiates by applying equal weights to all training samples and constructs an initial weak learner [41]. In successive iterations, events that are misclassified are assigned higher weights, while the number of correctly classified samples decreases, thereby directing the learning process toward the more difficult cases.

Through this iterative reweighting mechanism, AdaBoost incrementally improves the overall model by emphasizing the importance of errors made in previous rounds. By reducing the influence of accurately predicted observations and amplifying that of incorrectly predicted ones, the algorithm systematically refines the decision boundary. The overall classification is determined through a weighted majority vote of all weak learners, where each model’s contribution is proportional to its predictive effect [42], [43].

RF is one of the ensembles learning algorithms in ML, based on the principle of constructing multiple decision trees [44]. Due to its resistance to overfitting and its ability to deliver high generalization performance, the RF algorithm is frequently preferred by researchers. It is applied for both classification and regression applications. In regression tasks, the final prediction is obtained by averaging the outputs of individual decision trees. In classification, each tree votes for a class, and the final prediction is determined by majority voting among the trees [45].

The RF algorithm builds decision trees, from randomly selected training samples, and aggregates their outputs using majority voting scheme to determine the outcome [46], [47]. This ensemble approach contributes to the consistency and reliability of the predictive model, particularly in complex prediction tasks involving high-dimensional or noisy data.

Grid Search Cross-Validation (GridSearch CV) is a widely used technique for identifying the optimal hyperparameter values in ML models [48]. It is primarily employed to optimize model performance by systematically evaluating multiple parameter combinations. GridSearch CV enhances the reliability of ML models by integrating cross-validation into the optimization process [49].

In this method, the hyperparameter optimization process begins with the definition of a broad search space and parameter range. Subsequently, based on previous research findings and feedback from model performance, the search space and step size are progressively refined. This iterative procedure continues until the most suitable combination of hyperparameters is achieved.

In recent years, SHAP, has gained significant attention among researchers as a reliable approach for interpreting the outputs of AI algorithms. SHAP allows the model to be interpreted for a more detailed analysis within classification methods. SHAP algorithm stands out for its ability to fairly and systematically calculate the role of each feature in determining the model outcome, based on a game-theoretic approach [50], [51]. Conceptually, the Shapley value is structured around two key components: the game and the players. In this context, the game refers to the prediction model, while the players represent the features (i.e., model parameters) [52]. The Shapley value is used to quantify the contribution level of each feature to the model’s outcome.

SHAP's powerful visualization capabilities enable decision-makers to more clearly understand how the model responds to different variables, thereby enhancing model interpretability and promoting its broader adoption. Analyses typically close to interpretation between input and output models become interpretable with SHAP [53], [54].

When interpreting outputs using SHAP, the SHAP value plays a crucial role. In SHAP, the output of the forecasting model for a single observation value "$x$" is explained by a linear function, as defined in Eq. (3).

where, $x$ is the state to be interpreted, $x^{\imath}$ is the input, $\emptyset_i$ is the base value missing in all inputs, and $M$ is the number of input features [55]. This method can be used to explain the most contributing factors [11].

In this study, the performance of the ML models was determined via multiple evaluation metrics, namely accuracy, Cohen’s Kappa, precision, recall, F1-Score, ROC curve, and AUC. These metrics were selected to offer a detailed assestment of the models’ overall accuracy, class-specific performance, and ability to distinguish between different outcome classes.

Accuracy is a fundamental performance metric that represents the proportion of the total number of correctly predicted observations to the overall number of observations, and it is calculated using Eq. (4) [56].

Recall evaluates the ability of a classifier in correctly identifying positive cases, reflecting the proportion of true positives among all actual positives. The formula for calculating recall is provided in Eq. (5) [57].

Precision is used as a performance metric to minimize the number of false positives, indicating the proportion of correctly predicted values that are true positives, as shown in Eq. (6) [58].

Here, TP, TN, FP, and FN denote true positive, ture negative, false positive, false negative, respectively.

The F1-score provides a balanced evaluation of precision and recall by calculating their harmonic meaning, as expressed in Eq. (7) [59].

Cohen's Kappa coefficient serves as a statistical tool to evaluate the degree of agreement between two raters, considering the possibility of chance agreement. Essentially, it quantifies how much the observed agreement exceeds what would be expected by random chance when ratings are made on a nominal scale. Cohen's Kappa measures the accuracy of classifiers in statistics, psychology, biology, and medicine. Cohen's Kappa is calculated by Eq. (8) [60].

$P_0$ denotes of overall agreement or accuracy, and $Pe$ is the probability of agreement by chance [61]. Cohen recommended interpreting Kappa values as follows: A value of $\leq$ 0 indicates no agreement, 0.01–0.20 indicates slight agreement, 0.21–0.40 corresponds to fair agreement, 0.41–0.60 suggests moderate agreement, 0.61–0.80 reflects substantial agreement, and values ranging from 0.81 to 1.00 indicate perfect agreement [62]. The ROC curve is a widely employed tool for assessing the effect of classification models across a range of threshold values. It is generated by representing the true positive rate (TPR) against the false positive rate (FPR), with the x-axis representing the FPR and the y-axis corresponding to the TPR [63]. By connecting the threshold-specific points for each TPR–FPR pair, the ROC curve offer a graphical illustration of the trade-off between sensitivity and specificity under varying decision thresholds [64]. Moreover, it captures the functional relationships among competing alternatives, thereby supporting decision-making under uncertainty [65].

The AUC method is utilized in classification applications to determine the impact of model performance. AUC is defined as the area under the ROC curve, which is displayed on the ROC graph [66]. It is evaluated in conjunction with the ROC curve in studies [67]. The AUC value ranges from 0 to 1 [68]. As the value approaches 1, it indicates that the model has a higher discrimination potential. The higher the AUC, the better the model accurately predicts each class [69]. The AUC area is given according to the relationship between FPR and TPR in the Eq. (9) [66]. The AUC and ROC curves are shown in Figure 2.

Here, $x1$ and $x2$ represent the FPR range, and $y$ represents the TPR value.

In this study, the workflow is illustrated in Figure 3. It begins with data collection, where relevant parameters are gathered as reported by [21]. The values of LOS are defined as 0 (A), 1 (B), 2 (C), 3 (D), and 4 (E). The collected data then undergoes preprocessing, which includes cleaning, normalizing, and structuring to prepare the dataset for analysis.

Following data processing stage, the dataset is seperated into training and test subsets. The training data is used to develop predictive models for LOS classification by applying several ML algorithms, such as KNN, SVC, RF, and AdaBoost.

The test set is subsequently used to evaluate the performance and generalizability of these models through metrics including accuracy, precision, recall, AUC, and ROC. Cohen’s Kappa coefficient is computed to assess the agreement between predicted and actual LOS categories beyond chance. Finally, the Shapley method is employed to interpret the contribution of each input feature to the model outputs, enhancing the explainability and transparency of the ML predictions.

In this study, various ML models were developed for LOS prediction, aiming to analyze the data and enhance predictive accuracy. To ensure the effectiveness of the models and the reliability of the results, relationships among the input variables were thoroughly examined in advance using a correlation matrix (Figure 4). The analysis revealed that the number of cars, hourly volume, and flow rate exhibited a strong positive correlation with LOS, whereas the number of trucks showed a weak negative correlation. Based on these findings, ML models were implemented using KNN, SVC, RF, and AdaBoost algorithms. Prior to model training, the dataset was split into two parts: 80% training and 20% testing. To ensure robust hyperparameter tuning and minimize the risk of overfitting due to the relatively small dataset, a 5-fold CV strategy was used on the training subset. GridSearch CV was applied to systematically explore the hyperparameter spaces for each model (KNN, SVC, RF, AdaBoost). A suitable hyperparameter combination was selected according to the average CV accuracy, and the selected model was then refitted on the whole training set and evaluated on the external test set. The CV and test results of the ML models are summarized in Table 3, and the optimal hyperparameters are presented in Table 4.

Table 3 shows that the KNN model achieved the most balanced and robust performance with an average CV accuracy of 0.90 and a maximum test accuracy of 0.94. It also achieved a Cohen Kappa value of 0.92 for the test set. This indicates a strong agreement between the predicted and actual classes. The AdaBoost model provided the highest CV accuracy (0.92), achieving excellent fit for the training set. However, the test accuracy dropped to 0.88, and the Cohen Kappa value was 0.84. RF showed similar test performance (accuracy = 0.88, Kappa = 0.84), but its performance was lower than AdaBoost due to its lower CV accuracy (0.85). SVC, on the other hand, showed the lowest performance across all metrics. The CV accuracy is 0.83 and the test accuracy is 0.81. Cohen's Kappa value is 0.76. Overall, KNN emerges as the most reliable model in terms of both training and generalization performance.

The confusion matrices of the KNN model for the training and test sets are presented in Figure 5 and Figure 6, respectively. These matrices illustrate classifier’s performance by displaying the number of correct and incorrect predictions for each class. In the training set (Figure 5), the model demonstrates high accuracy, particularly for Class 0 and Class 2, with 24 and 14 correct predictions, respectively. Only a few misclassifications are observed, suggesting effective learning of the underlying patterns in the training data. This finding aligns with existing studies in literature, where the KNN algorithm has been shown to perform well in classification problems due to its simplicity and strong local decision-making capabilities [70], [71].

In the test set (Figure 6), the KNN model maintains strong generalization performance, achieving perfect classification for Classes 0 through 3 and only a single misclassification in Class 4, which was predicted as Class 3. The clear diagonal dominance in both matrices highlights the model’s robustness and its ability to perform reliably on unseen data.

Additionally, key performance metrics, including precision, recall, F1-score, accuracy, macro average, and weighted average, were calculated for each class and are summarized in Table 5. These metrics offer a detailed assessment of the model’s predictive performance across all classes.

According to Table 5, the KNN model achieved the highest accuracy, recall, and F1 score values for classes 0, 1, and 2. Accuracy, recall, and F1 score were recorded as 1.00 for these classes. Additionally, when analyzing the weighted average metric, the model achieved a precision of 0.89, a recall of 0.94, and an F1-score of 0.91. These results demonstrate that the KNN model is effective in LOS estimation.

Furthermore, ROC curves were generated for both the training and test sets of the KNN model, and the corresponding AUC values were calculated to assess its classification performance. As illustrated in Figure 7 and Figure 8, the model achieved high AUC scores on both datasets, indicating a strong ability to distinguish between LOS classes. Additionally, micro-average and macro-average ROC curves were computed. While the micro-average considers each prediction equally and reflects overall performance, the macro-average treats all classes equally by averaging the AUCs of each class. These curves provide a broader understanding of the model’s effectiveness. The obtained results indicated that the KNN model achieves good performance across the datasets, demonstrating its reliability and robustness for practical use.

Figure 7 shows the ROC curve for the training set of the KNN model. Notably, AUC for Class 0, 2 and 4 is 1.00, indicating perfect prediction for this class during the training phase, while the AUC values for the other classes are above 0.99. Figure 8 presents the ROC curves and corresponding AUC values for the test set. The AUCs for Classes 0, 1, and 2 are all 1.00, demonstrating successful classification of these classes. Meanwhile, the AUC for Class 3 is 0.96, and for Class 4, it is 0.93.

To provide deeper insight into the results obtained from the KNN model, SHAP values were employed to clarify the relationship between the input features and the model's predictions. SHAP values quantify the role of each feature to the prediction results, allowing for a more interpretable analysis of the model's decision-making process, as also widely demonstrated in the literature to enhance model transparency and explainability across various domains [72]. These values were meticulously calculated, and a SHAP summary plot, presented in Figure 9, was generated to visually represent the impact of individual features across the dataset. This visualization enables a clearer understanding of which features most strongly influence the model’s outputs, thereby enhancing transparency and interpretability of the predictive model.

Figure 9 presents the effect of each input feature on the predicted classes. The flow rate parameter shows the greatest impact on Classes 0 and 2, while its influence is minimal on Class 4. Similarly, the hourly volume and number of cars exhibit the highest impact on Class 0. Overall, flow rate, hourly volume, and number of cars appear to significantly influence all classes. In contrast, the number of trucks has a noticeable effect only on Classes 0 and 1.

This study developed and evaluated four ML models: KNN, SVC, RF, and AdaBoost for predicting the LOS. Correlation analysis revealed strong positive relationships between LOS and the number of cars, hourly volume, and flow rate, while the number of trucks showed a weak negative correlation. The efficiency of the ML models was optimized using GridSearch CV for hyperparameter tuning. Key findings are summarized as follows:

$\bullet$ KNN and RF models achieved near-perfect performance, outperforming SVC and AdaBoost.

$\bullet$ The GridSearch CV technique, combined with 5-fold CV, effectively optimized the model performance and prevented overfitting.

$\bullet$ The KNN model achieved the highest accuracy (0.94) and Cohen’s Kappa coefficient (0.92) on the test set.

$\bullet$ Precision, recall, and F1-score metrics confirmed robust class-wise performance of the KNN model (for class 0: precision 1.00, recall 1.00, F1-score 1.00). The weighted-average precision, recall, and F1-score were 0.89, 0.94, and 0.91, respectively.

$\bullet$ ROC curve analysis demonstrated high discriminative ability, with AUC values close to or equal to 1 for most classes in both training and test sets.

$\bullet$ SHAP analysis identified flow rate as the most influential input feature on KNN model predictions, while the number of trucks was the least influential.

$\bullet$ Flow rate, hourly volume, and number of cars affected all LOS classes, whereas the number of trucks influenced only classes 0 and 1.

These findings demonstrate the effectiveness and reliability of ML approaches, particularly KNN, for accurate LOS prediction.

As a conclusion of this study, it can be said that ML systems and analysis techniques can be effectively used to predict the LOS behavior of traffic flow. Especially, the KNN method provided better results in this analysis. However, other models such as SVC, RF, and AdaBoost also demonstrated considerable predictive capabilities, highlighting that the choice of method may depend on specific data characteristics and application requirements. Additionally, interpretability tools like SHAP analysis helped to identify key input variables influencing predictions, emphasizing the importance of combining accurate modeling with explainable AI approaches for reliable and comprehensive traffic flow assessment.

Limitations and future work:

This study provides an initial exploration of LOS prediction using ML techniques on a small-scale campus road dataset. However, the data set comprised only 77 observations collected from a single road segment, which limits the statistical representativeness of the findings. Previous studies have also indicated that small datasets can limit model generalization, although they remain useful for methodological feasibility analyses. The findings from this preliminary study are intended to guide future research that will include more diverse and larger-scale datasets, ensuring improved external validity over time. Future work will involve collecting larger and more diverse datasets from various road types and seasons, incorporating additional variables such as weather, driver behavior, and intersection control types to improve generalization.