8. Efficiency & Bandwidth
While Kafka advertises itself as real-time, it is more accurately described as “near real-time”. This is due to several confounding factors, such as pipeline delays and Broker replication. These tradeoffs ensure that no brokers miss information and that all brokers stay in sync with each other. While you can have latency in the 5 to 50 ms range under normal parameters, Kafka themselves state that they focus on throughput efficiency over super low latency.1
With a system that has 2 vCPU and 4 Gigabytes of RAM with an SSD, the throughput is approximately 15,000 messages per second per broker, with a latency of about 15ms. That is a total of about 45,000 messages per second. Using ECS and its maximum size of 16 vCPU and 128GiB of RAM would provide upwards of 100,000 to 200,000 messages per second per broker. We made our Kafka brokers inherently scalable, so the number of brokers and controllers can always be scaled up. This means that we could easily scale to 50 total brokers before running into significant issues. Specifically, the system latency, network bandwidth, and disk speed will become the bottlenecks, which simply increasing the number of brokers will not fix.2
Note that a difference in message size can make a sizable impact on data flow. For testing, we used a record size of 500 bytes with a batch size of 1 byte, as that is the default size recommended by Red Hat for batch sizes. As we are using at least one semantics, the lead broker replicates messages to all available follower brokers in order to ensure acknowledgment of message delivery. This has an impact on overall data delivery, specifically a slight reduction in speed while providing no dropped messages.
With that in mind, we can use the built-in benchmark tool Kafka-producer-perf-test with a cluster of 3 nodes. This tool is available at opt/Kafka/bin/Kafka-producer-perf-test.sh3
