Queues, Streams, Eventing
Kafka partitions: multiple-choice review
Crux Multiple-choice synthesis across the Kafka partitions unit — key hashing, consumer-group assignment, the one-way door of partition count, skew, and rebalancing.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each mirrors a call you make in a real Kafka design review — not a definition to recite, but a tradeoff to weigh before you touch a topic config.
Goal
Confirm you can connect key hashing, consumer-group assignment, the one-way door of partition count, hot-partition skew, and rebalancing — the synthesis the lesson built toward.
Quiz
Completed
A topic has 8 partitions and a consumer group of 12 members, yet lag keeps climbing. What is the real constraint, and what actually fixes it?
Heads-up There is no contention to lock — each partition is assigned to exactly one consumer in the group. The extras simply receive no assignment; they are not fighting over anything.
Heads-up No assignor lets two consumers in one group co-own a partition. One-consumer-per-partition is a fundamental rule of consumer groups, not an assignor option.
Heads-up Poll tuning helps a consumer that already owns a busy partition, but it cannot manufacture parallelism the partition count forbids — 4 consumers still have nothing assigned.
Quiz
Completed
An order-events topic is keyed by orderId on 6 partitions and lagging at peak. Per-order ordering must hold. Someone proposes raising it to 12 partitions in place. Why is that the trap, and what is the senior move?
Heads-up Kafka never moves existing records on a partition increase — they stay where they were written. The break is that future records for a key now hash elsewhere, so its history is split across two partitions.
Heads-up Kafka orders within a partition, never across a topic. Once a key's records land on two partitions there is no global order to fall back on, so per-key ordering is genuinely lost.
Heads-up You cannot decrease partition count at all — it only ratchets upward. There is no correct way to merge two ordered logs, so Kafka refuses; the change is effectively permanent.
Quiz
Completed
One consumer in a healthy group is pegged at 100% CPU and lagging while the other five idle. Partition and consumer counts are balanced. What is happening, and where is the fix?
Heads-up The partitions are evenly assigned — the imbalance is in the data, not the assignment. A rebalance reshuffles ownership but the hot key still funnels to one partition afterward.
Heads-up Consumer parallelism is capped at the partition count, and a single key still maps to a single partition. New consumers either idle or own other partitions — none can relieve the one hot key.
Heads-up Fetch tuning helps marginally but cannot fix a structural skew where one partition carries a disproportionate share of traffic. The bottleneck is one key on one partition.
Quiz
Completed
During a routine rolling deploy, consumption across an entire group halts for several seconds each time a pod restarts, even though only one consumer is replaced at a time. What is the cause, and what reduces it?
Heads-up The pause spans the whole group, not just the new pod, and it happens before any fetching — it is the rebalance protocol revoking and reassigning partitions, not a fetch-sizing issue.
Heads-up Consumer rebalances reassign partition ownership among consumers; they do not move partition data between brokers. Replica movement is a separate, broker-side operation.
Heads-up The stall is the rebalance protocol itself stopping consumption, not coordinator capacity. The fix is the rebalance strategy — cooperative rebalancing and static membership — not more brokers.
Quiz
Completed
A team plans a topic and asks whether to provision 8 partitions for current load or 64 for headroom. What is the right framing?
Heads-up Partitions are not free: each adds metadata, file handles, and replication overhead, and too many degrade throughput and lengthen controller failover. There is a real upper bound, not just an upside.
Heads-up Raising partition count later is a one-way door that rehashes keys and breaks per-key ordering. You provision headroom up front precisely to avoid the in-place resize.
Heads-up You can never decrease partition count, and increasing it has serious ordering consequences. It is one of the least reversible decisions in a Kafka design.
Quiz
Completed
A producer sends records with no key to a 6-partition topic, and a downstream consumer relies on processing events for the same entity in order. Why does this design fail, and what is the fix?
Heads-up Kafka orders only within a single partition, never across a topic, and never by wall-clock send time. Null-key records scatter, so same-entity events can interleave out of order.
Heads-up More partitions cannot pin a null-key record to a consistent one — without a key the partitioner has nothing to hash, so records keep scattering. The fix is to supply a key, not to add partitions.
Heads-up Idempotent producers prevent duplicate writes within a partition; they do not co-locate records that have no key. Ordering for an entity comes from keying it onto one partition.
Recap
The through-line is one mechanism wearing three hats: hash(key) % N does distribution, per-key ordering, and co-location at once. From it everything follows — ordering holds only within a partition, consumer parallelism is hard-capped at the partition count, and changing N rehashes keys and breaks per-key order, so the count is a one-way door you provision for up front rather than tune. The two production failures, hot-partition skew (fix at the key level) and stop-the-world rebalances (tamed by cooperative rebalancing and static membership), both trace back to the same lesson: the key and the count are the real design decisions, and they are made before the topic exists.