Consumer Groups
Scale ordered log processing by assigning partitions across cooperating consumers.
2–5 minute refresher
P0104105106107
P181828384
Consumer group A · offset 106Consumer group B · offset 82
One partition → one active consumer per group → durable offset
30second
refresher
refresher
A consumer group divides log partitions among members so each partition has one active consumer in that group. Independent groups maintain independent offsets and can interpret the same log differently.
What problem does it solve?
One consumer cannot process an unbounded event stream, but unconstrained parallelism breaks per-key ordering and duplicates work.
How it works
- Join the group with a stable member identity where possible.
- Assign each partition to one active member.
- Read records in partition order.
- Persist processed offsets after the durable outcome.
- On membership change, revoke, checkpoint, and transfer assignments safely.
Decision guide
Key trade-offs
| Choice | What you gain | What it costs |
|---|---|---|
| More partitions | More processing parallelism | Metadata, open files, and coordination |
| Frequent commits | Less replay after failure | More overhead |
| Sticky assignment | Warm caches and fewer moves | Slower correction of uneven work |
What happens if?
A consumer pauses beyond the session timeout
The group reassigns its partitions while the old member may resume. Generation or epoch checks must prevent stale commits, and side effects must tolerate replay from the last durable offset.
Where it appears
- Kafka consumer groups
- Pulsar subscriptions
- Kinesis applications
- Google Pub/Sub subscriptions
Senior interview modeWhy does adding consumers beyond the partition count not increase throughput?Show answer
Within one group each partition has only one active owner, so extra members have no partition to consume. More partitions or intra-message parallelism is required.