Kafka Consumer Re-balancing on Static Membership

The purpose of this post is to explain about Kafka consumer rebalancing and how static membership help to avoid consumer rebalancing. To get an idea about kafka partitions assignment for consumers, please refer this blog post.

Key Points

• Single consumer rebalancing will effect on whole group, so it will stop message processing in whole consumer group. 

• On consumer rebalancing, KafkaRebalanceListener#onPartitioned() method will be called just before it take away it's partitions. 

          Examples: 1. Consumer requesting leave group.

                            2. Already existed consumers partitions move to newly joined consumer.

• Both consumer leave and join cause for consumer rebalance.

• When you use static membership , coordinator will aware about that group_instance_id property since it will send to coordinator with request at join group process.

• With static membership, consumer doesn't send Leave Group Request in a failure or restart. Because of that broker coordinator will wait till it reach session.timeout.ms . If the consumer came back within that time period, there won't be any rebalancing.

 

Kafka Partition Assignment for Consumers

The purpose of this post is to explain about kafka partition assignment.

Key Points

•  If we have fix number of partitions and if we use messages keys in producer level, messages with same keys going to same partition.

• Producer only send messages to partition leader.

• It's not compulsory to have a key in a message, then broker will use round robin method to distribute messages among partitions 

• Kafka guaranteed that a topic partition is assigned only to one consumer within the group.

• All the consumers will send a join group request to coordinator broker, but first consumer will be the leader of the same consumer group.

• Then consumer leader send synch group request with a body but rest of the consumers will send empty synch request to coordinator broker.

• Consumer joining request body contains session.timeout.ms , max.poll.interval.ms .

• Consumer leader is responsible for partition assignment within rest of the consumers in same group locally.

• When consumers synching only consumer leader send request with a body while rest of the consumers in the same group will send empty request.

• When consumers synching, all the consumers get a response with member assignment details.

• Once coordinator broker responds for all the SynchGroup requests, consumer's configured listener will call onPartitionAssigned() method.

• All the consumers periodically (heartbeat.interval.ms) send heart beat request to coordinator.

• If coordinator triggers a rebalance, other consumers will only know of this by receiving the heartbeat response with REBALANCE_IN_PROGRESS exception encapsulated. Then only consumers know that they need to rejoin the group.

References:

https://stackoverflow.com/questions/43991845/kafka10-1-heartbeat-interval-ms-session-timeout-ms-and-max-poll-interval-ms

https://medium.com/streamthoughts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications-e94baf68e4f2

https://medium.com/analytics-vidhya/the-introduction-to-kafka-topics-and-partitions-ec97dd7ecfbe