A Comparative Study Evaluation of Kafka and RabbitMQ: Performance, Scalability and Stress Test in Distributed Messaging Systems

Abstract

The two most widely used Message-Oriented Middleware (MOM) technologies are Apache Kafka and RabbitMQ, both of which have fundamental differences in terms of architecture and performance characteristics. Kafka is designed for high-throughput and good scalability data stream processing, while RabbitMQ excels in message routing flexibility, delivery reliability, and complex queue management. This study presents a comprehensive comparative analysis of two leading message brokers, namely Apache Kafka and RabbitMQ to evaluate performance, scalability, and behaviour under stress test, for the selection of the most suitable message broker in modern distributed system architectures. The experimental testing process was carried out in four different scenarios: message size variation of 1 KB, 10 KB and 100 KB aimed at measuring performance based on payload size, message volume variation of 10,000, 50,000 and 100,000 messages to see throughput limits and resource usage, consumer number variation of 1, 5 and 10 Measuring the scalability of the consumer system, then a high-intensity pressure test of 100,000 messages in 10 seconds to evaluate the stability and latency of the overload. Key performance metrics, such as throughput, latency, CPU usage, and RAM consumption are carefully evaluated. The overall results of the experiment were more suitable for systems that affect the speed and volume of messages, while Kafka was more appropriate for extreme workloads with high durability requirements. This experiment provided empirical data concluding that RabbitMQ is highly effective for applications that require sending high-volume, low-latency individual messages, while Kafka's strength lies in handling specific data stream sizes and maintaining stability under intense and sustained loads.