Evaluating Data Sources and Datasets in Intelligent Transport Systems Through a Weighted Scoring Model

Intelligent Transportation Systems (ITS) integrate advanced technologies and communication systems into the transportation infrastructure and vehicular fabric, aiming to bolster safety, mobility, and efficiency. ITS applications are engineered to enhance transportation performance by mitigating crash occurrences [1, 2], augmenting roadway visibility [3], alleviating congestion [4], reducing accident severity [5, 6], and optimizing fuel efficiency. These systems encompass intelligent solutions applied across all vehicular operation phases to realize the vision of safer and more efficient roadways.

Presently, ITS implementations are prevalent within urban centers and along highways, undergirded by an array of monitoring devices, including cameras, unmanned aerial vehicles (UAVs), light detection sensors (LIDAR), radar, and ultrasonic sensors. It is through these devices that critical data on driver behavior—encompassing acceleration, braking, lane changing, and speed—are harvested under both normal and adverse conditions.

At the core of ITS lies an extensive reliance on datasets, amassed through varied data collection methodologies, to catalyze a transformative shift in transportation paradigms. The potency of data is harnessed to elevate the operational efficiency, safety, and environmental sustainability of transportation networks.

Datasets constitute the foundational element of ITS, encapsulating crucial information on traffic flow, road conditions, user behavior, and environmental variables. By analyzing these datasets, ITS are empowered to decode the intricacies of transportation systems, thereby enabling informed decision-making and the deployment of intelligent responses.

Data acquisition techniques in Intelligent Transportation Systems (ITS) are crucial for the procurement of pertinent data, utilizing an array of methodologies including sensor technologies, imaging devices, and aerial surveying by drones. Ground-based sensors, strategically deployed along transportation arteries, are responsible for the real-time capture of traffic metrics such as volume, velocity, and congestion levels. Additional insights into roadway conditions and traffic dynamics are procured via vehicular and infrastructural cameras and sensors. Unmanned Aerial Vehicles (UAVs), or drones, offer a vantage point for aerial surveillance, further enriching the data landscape.

The synthesis of multifaceted datasets with advanced collection mechanisms forms the backbone of ITS, synergistically enhancing the intelligence and efficacy of modern transportation systems. These integrated datasets and collection sources are pivotal in steering transportation towards a more intelligent, efficient, and sustainable future.

Central to the discourse of this paper are several inquiries: Which methodologies are employed for the gathering of data to compile ITS datasets? Which datasets are considered preeminent within the ITS field? And crucially, how can these disparate methodologies and datasets be effectively compared?

To address these inquiries, the present study adopts the Weighted Scoring Model (WSM) to conduct two distinct comparative analyses. Initially, the comparison of data source methods—including drones, sensor-equipped vehicles, simulators, and infrastructure-based sensors—is undertaken. Subsequently, the focus shifts to the evaluation of datasets currently utilized in ITS research. Criteria for comparison are meticulously delineated, encompassing scenario depiction, naturalistic behavior capture, efficiency, flexibility, duration of monitoring, and error frequency for data collection methods. For datasets, essential parameters such as mapping detail, temporal resolution, feature richness, data provenance, and user typology are established. Following the establishment of these criteria, the WSM methodology is detailed and applied as delineated in Section 4. Results from the WSM analysis are subsequently presented in a spider graph format, providing a visual comparison of each data collection method and dataset against the defined criteria.

The structure of this paper is as follows: Section 2 elucidates the most significant datasets and data collection methods utilized in ITS. Section 3 presents a comparative analysis of these methods and datasets. Section 4 introduces the WSM methodology and outlines the research methodology. Sections 5 and 6 apply the WSM approach to evaluate the data collection methods and datasets respectively, using weighted attributions to compute and compare final scores. The paper concludes with a discussion of the findings and future perspectives in Section 7.

In this section, we present data sources used to collect datasets in ITS. Then we present datasets collected to solve ITS problems. Using drones as sensors for traffic monitoring, then existing datasets for onboard sensors and driving simulators.

To collect datasets, a range of innovative methods are employed. Drones equipped with cameras and sensors are deployed to capture aerial views and collect data on traffic patterns, road conditions, and infrastructure monitoring. In addition, ground-based sensors installed along roadways provide real-time information on traffic volume, speed, and vehicle classification. Driving simulators allow researchers and developers to generate simulated environments, enabling them to study driver behavior, test algorithms, and evaluate new transportation strategies.

The use of camera-equipped drones to measure every vehicle’s position and movements from an aerial perspective is a novel approach that has the potential to revolutionize the way traffic flow is monitored and managed. By having a continuous, real-time bird’s eye view of traffic, bottlenecks, and congestion can be identified and addressed more quickly and effectively. Additionally, this data can be used to study driver behavior and create alert systems in vehicles and to police systems to make the necessary decisions. Also, to optimize traffic patterns and road safety. Sensors on series-production vehicles are used to measure the vehicle´s environment and collect the data [7]. The data collected by the sensors can be used to improve the safety and efficiency of the vehicle and driver, it can be used to provide safety warnings to the driver in the form of visual, auditory, or haptic feedback. The sensors can also be used to monitor the health of the vehicle and its components. The installation of infrastructure sensors at dedicated masts or streetlights located along road segments can permanently monitor a certain road segment for signs of wear and tear. This is especially useful for detecting changes in road conditions like the flux of traffic and detecting abnormal driver behavior that could potentially lead to accidents. By constantly monitoring the condition of the road, these sensors can help to improve the safety of drivers and passengers alike. A simulator of conduits can be used to collect datasets for a variety of purposes. It can be used to collect data on the performance of a system, or to collect data for research purposes. Additionally, a simulator of conduits can be used to collect data for educational purposes or to collect data for marketing purposes.

A key component of ITS is the availability of diverse datasets that enable the system to tackle transportation challenges effectively. These datasets encompass real-time traffic information, weather conditions, road infrastructure details, vehicle data, and user behavior patterns. They provide crucial insights for addressing congestion, optimizing routes, and predicting traffic flow.

Drones equipped with high-resolution cameras can record traffic from a so-called "bird's-eye view" with high position precision. We present the most popular datasets dedicated to ITS. The Stanford Drone Dataset [8] was the first dataset with the trajectories of several road users that was created from the point of view of a drone. It is publicly available and was published in 2016. It is suitable for the analysis of the behaviors and interactions of pedestrians. It consists of nine hours of data from 8 locations on the Stanford campus. The dataset includes 10,300 pedestrian, bicycle, automobile, skateboard, cart, and bus trajectories. Only around 7% of the targets in the sample that have been tagged are cars, compared to a large ratio of identified bikes and pedestrians. The highD dataset [7], which was published in 2018, is the first extensive naturalistic vehicle trajectory dataset on German highways using drone-captured video data. The observations were conducted at six separate locations and involved 110,000 vehicles traveling 45,000 kilometers in 16.5 hours for the highD dataset. The CITR and DUT, two drone-based datasets, were published in 2019 [9]. The dataset, which lasted for less than 30 minutes, was centered on investigating pedestrian behavior when interacting with cars. The controlled experiment used to create the CITR dataset took place in a parking lot, in contrast to the DUT dataset, which comprises pedestrians' naturalistic, uninstructed trajectories. The INTERACTION dataset [10] is a dataset that was produced utilizing drones and includes the realistic motions of numerous traffic participants. Several highly interactive driving scenarios are included in the collection, which comes from China, Bulgaria, Germany, and the United States. It contains measurements from 11 locations and the recording time is up to 16.5 hours. The dataset offers HD-map data in lanelet2 format for the first time. In 2020, the inD dataset [11], which was captured at four various unsignalized junctions in Germany, was published. Over the course of 10 hours, it contains a total of 13,599 trajectories. The inD dataset divides all users of the road into four categories: cars, trucks or buses, bicyclists, and pedestrians. Another urban dataset named the rounD dataset [12] has been published in 2020; it contains over 13,746 trajectories recorded over six hours at three different locations, unsignalized roundabouts in Germany. The openDD dataset [13] is collected in Germany in 2020. openDD contains 84,774 trajectories in 62 hours and HD map data of seven different unsignalized roundabouts. At the signalized intersection in China, a drone dataset SIND [14] was collected and published in 2022. SIND includes traffic light states and HD maps, which contain 7 hours of recording including 13,248 trajectories and include 7 road user types: cars, trucks, buses, tricycles, bikes, motorcycles, and pedestrians. The trajectory dataset called as CitySim dataset [15] was published in 2023 and was taken from drone videos. CitySim has vehicle interaction trajectories extracted from 19 hours at 12 different locations. More severe and significant critical safety events are present in CitySim dataset, which offer supportive scenarios for safety-focused research. The Driving Behavior Net (DBNet) [16] is a dataset for driving behavior research. It includes aligned video, point cloud, GPS and driver behavior (speed and wheel). The dataset is collected in 2018.

The most widely used vehicle motion dataset in the behavioral research fields is the Next Generation Simulation (NGSIM) dataset [17]. Cameras positioned on buildings gathered the raw data, which was then automatically processed. NGSIM has been registered in four different locations: Peachtree Street in Atlanta, Georgia; Lankershim Boulevard, located in Los Angeles, California; eastbound I-80 in Emeryville, California; and U.S. Highway 101 in Los Angele. In the Five Roundabouts Dataset [18], which was published in 2019, over 23 000 vehicles at five unsignalized roundabouts in Australia were followed using a total of six Ibeo LIDAR scanners onboard a vehicle parked close to the roundabouts, yielding more than 60 hours of data. The Strategic Highway Research Program 2 (SHRP 2) NDS [19] database includes data from 50 million vehicle miles and 5.4 million trips, SHRP 2 was collected by 3,147 volunteers using radar, raw-video, and video of the driver at 6 different sites in the United States: central Indiana; Erie County, New York; Tampa, Florida; Durham, North Carolina; central Pennsylvania; and Seattle, Washington. The 100-Car Naturalistic Driving Study dataset [20] contains several examples of excessive driver behavior and performance, like extreme weariness, impairment, mistakes of judgment, risk-taking, aggressive driving, and traffic violations. The collection contains data from a very competent instrumentation system, including 5 channels of video, various vehicle statuses, and kinematic sensors. It also contains data from roughly 2 million vehicle miles and almost 43 thousand hours of data. The European Commission is the founder of the UDrive [21], a large naturalistic driving study in Europe. More than 1,200 drivers contributed the information on more than 35 million kilometers driven in UDrive dataset. The information includes raw video, GPS position, onboard CAN-bus records, front-facing radar, and camera images. However, the datasets UDrive and SHRP 2 are not freely available to the public. The driver behavior dataset [22] is gathered across four car excursions that last, on average, 13 minutes each, using a smartphone in 2017. The Honda Research Institute Driving Dataset (HDD) [23] was published in 2018. The dataset comprises of 104 hours of real human driving in the San Francisco Bay Area, the data was collected using a vehicle fitted with various sensors. The purpose of this dataset is to study driver behavior in real-life environments. The UAH-DriveSet [24] is a dataset that was gathered from six different drivers and cars and is used for the analysis and classification of driving behavior. Three unique driving behaviors were included in the data: normal, drowsy, and aggressive.

According to some researchers, it is challenging to directly model using equations the interactions between human drivers. In order to solve this issue, simulations like CARLA [25] developed by researchers at Intel, and AirSim [26], developed by Microsoft, are examples of such simulators that are both open-source. may more easily imitate a human driver's behavior thanks to learning-based methods for characterization of human-driver behavior. The HRI Driver Behavior Dataset (HDBD) [27] contains driver behavior collected using simulator and real scene videos from 32 participants. Each participant recorded 4 sessions, each consisting of 10 intersections that last approximately eight minutes.

In this section, we will compare the data sources and datasets discussed earlier based on several characteristics.

Multiple Criteria Decision Making is a subset of operations research dedicated to assessing and comparing various options or alternatives using multiple criteria or factors. It includes a variety of methods, like Weighted Scoring Model (WSM), Analytic Hierarchy Process (AHP), Analytic Network Process (ANP), etc. In this paper, we used the WSM [29] based on its simplicity and flexibility to compare data sources and datasets. In this approach, each criterion is assigned a weight, signifying its relative importance in the decision-making process. Subsequently, each option is evaluated and scored against these criteria. To calculate a weighted score for each option, the WSM multiplies the score of each criterion by its corresponding weight and then sums up these weighted scores. This process enables decision-makers to quantitatively analyze and rank the options, considering both the significance of each criterion and the performance of each option against those criteria. The WSM Method is utilized in this paper to compare the data source and dataset used in Intelligent Transport Systems (ITS). The application of this strategy involves the following steps:

Determine criteria: Firstly, the criteria that constitute the data collection methods are identified. These criteria serve as the basis for evaluating and comparing the data sources and datasets.

Assign weight to the criteria: Each criterion is assigned a weight that reflects its relative importance in comparison to the other criteria. The weights are determined based on the significance of each criterion in achieving the research objectives.

Create a table of criteria and measurement methods: A table is constructed, listing the chosen criteria and the corresponding measurement methods used to assess the data sources and datasets.

Table of weight: Next, a table is created that displays the assigned weights for each criterion. The scores indicate how well each element performs with respect to each criterion.

Calculation of method score: The WSM calculates a weighted score for each criterion in data source and dataset by multiplying the score of each criterion by its assigned weight and then summing them up. This results in an overall score for each element. The element with the highest weighted score is the one that you should choose.

In this paper, the Weighted Scoring Model (WSM) method is applied at the data source level to determine the best method for data collection. The WSM method is further applied to choose the best dataset among the datasets presented in section 2. Figure 7 illustrates the step-by-step process of this approach.

The paper initiates a comparative analysis of various data sources and datasets within the realm of ITS. Furthermore, it presents a comparative study employing a Weighted Scoring Model. It involves assigning weights to various criteria or factors that are relevant to the comparison of data sources and datasets. These criteria may include scenario description, naturalistic behavior, efficiency, flexibility, monitoring duration, and mistakes for data sources. On the other hand, criteria such as maps, hours, features, data source quality, and road user type are used to compare datasets. Each data source and dataset are then evaluated and scored against these criteria, considering their respective weights. The WSM calculates a weighted score for each data source and dataset, representing its overall performance based on the specified criteria. The results indicate that the drone method is the best measurement method to collect a dataset for the driver and vehicle, with a total score of 3.36. Additionally, the SinD dataset receives the highest score of 3.4. These models of WSM provide a quantitative and systematic approach to objectively compare data sources and datasets in the context of ITS, aiding decision-making processes and facilitating the selection of the most suitable data source and dataset for a given application. While this study is certainly important, it is worth noting that the choice of a dataset or data source also depends on various conditions, choices, and possibilities available to researchers. In future work, we explore the potential of machine learning and deep learning algorithms to study drivers' behavior.