Kafka のリバランスプロトコルについて、聞きたくても聞けなかったことすべて

Kafka のリバランスプロトコルについて、聞きたくても聞けなかったことすべて

Watch Now

Apache Kafka® is a scalable streaming platform with built-in dynamic client scaling. The elastic scale-in/scale-out feature leverages Kafka’s “rebalance protocol” that was designed in the 0.9 release and improved ever since then. The original design aims for on-prem deployments of stateless clients. However, it does not always align with modern deployment tools like Kubernetes and stateful stream processing clients, like Kafka Streams. Those shortcomings lead to two major recent improvement proposals, namely static group membership and incremental rebalancing.

This talk provides a deep dive into the details of the rebalance protocol, starting from its original design in version 0.9 up to the latest improvements and future work.

We discuss internal technical details, pros and cons of the existing approaches, and explain how you configure your client correctly for your use case. Additionally, we discuss configuration tradeoffs for stateless, stateful, on-prem, and containerized deployments.

Speakers

Matthias J. Sax

Matthias J. Sax, Software Engineer, Confluent

Matthias is a Kafka PMC member and software engineer at Confluent, working mainly on Kafka’s Stream API. Prior to Confluent, he was a PhD student at Humboldt-University of Berlin, conducting research on the data stream processing system. Matthias is also a committer at Apache Flink and Apache Storm.

サイトのご利用状況を把握しユーザーエクスペリエンスの改善へとつなげるため、当サイトでは Cookie (クッキー) を使用しています。Cookie についての詳細を確認するには、また Cookie 設定の変更をご希望の場合は、こちらをクリックしてください。閲覧を続行することにより、当社の Cookie 使用に同意されたものとみなされます。