Parallel Memory-Based Collaborative Filtering for Distributed Big Data Environments

The amount of information produced about any item or user has reached a very staggering level. Not only the volume of data, the velocity of data has reached an unprecedented magnitude. For any information retrieval or information processing system to work efficiently, it should be able to process massive amounts of data in real-time. Modern systems face a lot of challenges in managing data with high volume and velocity, especially when these systems are required to generate accurate predictions in a timely fashion. The most efficient way to ensure that modern information retrieval systems can adapt to the current volume and velocity of data is to implement them over a parallel and distributed environment. In this paper, we put forward a method for enhancing the scalability and performance of recommender systems in big data environments. By using the Euclidean distance to calculate the cosine similarity we introduce a technique which is efficient in parallelizing the algorithm for distributed environments. Thereby improving the computational efficiency and scalability of the recommender system. This enables such systems to manage large datasets with high accuracy and speed. With the help of parallel processing, our method can assist modern information retrieval systems keep up with the pace of ever-growing demands of data velocity and volume, ensuring real-time performance and robust scalability.