Measuring and minimizing latency in a Kafka Sink Connector

Apache Kafka is a popular choice for real-time data streaming due to its scalability, fault tolerance, and low latency capabilities. Kafka Connectors act as bridges between event-driven and non-event-driven technologies, enabling the streaming of data between systems. However, traditional approaches to scaling Kafka Connectors often prioritize throughput over latency optimization, which can negatively impact applications that rely on low latency for success. To address this issue, Ably developed its own Kafka Connector and conducted research to determine the optimal configuration for achieving minimal latency under moderate load scenarios. The company used distributed tracing with OpenTelemetry to measure end-to-end latency and identify bottlenecks in their system. By implementing a comprehensive approach using distributed tracing, they were able to pinpoint latency bottlenecks and understand the impact of various optimizations on overall performance. The research revealed that increasing the number of partitions did not significantly improve latency, while adjusting the tasks.max setting could help balance concurrency and resource utilization. Using simpler converters like the built-in String converter reduced overhead associated with serialization and deserialization. Experimenting with batching intervals showed that even small batching intervals increased latency, so avoiding batching altogether was the most effective approach to maintain low latency in their specific use case. Increasing the number of threads in the connector's internal thread pool significantly reduced latency by allowing more messages to be processed in parallel but came with trade-offs such as potential issues with message ordering. The insights and optimizations explored can be applied broadly to any Kafka Connector to improve performance and reduce latency, emphasizing the importance of understanding and optimizing built-in Kafka configuration settings and connector-specific settings.