The purpose of this post is to explain about kafka partition assignment.
Key Points
- If we have fix number of partitions and if we use messages keys in producer level, messages with same keys going to same partition.
- Producer only send messages to partition leader.
- It's not compulsory to have a key in a message, then broker will use round robin method to distribute messages among partitions
- Kafka guaranteed that a topic partition is assigned only to one consumer within the group.
- All the consumers will send a join group request to coordinator broker, but first consumer will be the leader of the same consumer group.
- Then consumer leader send synch group request with a body but rest of the consumers will send empty synch request to coordinator broker.
- Consumer joining request body contains session.timeout.ms , max.poll.interval.ms .
- Consumer leader is responsible for partition assignment within rest of the consumers in same group locally.
- When consumers synching only consumer leader send request with a body while rest of the consumers in the same group will send empty request.
- When consumers synching, all the consumers get a response with member assignment details.
- Once coordinator broker responds for all the SynchGroup requests, consumer's configured listener will call onPartitionAssigned() method.
- All the consumers periodically (heartbeat.interval.ms) send heart beat request to coordinator.
- If coordinator triggers a rebalance, other consumers will only know of this by receiving the heartbeat response with REBALANCE_IN_PROGRESS exception encapsulated. Then only consumers know that they need to rejoin the group.
No comments:
Post a Comment