Main Components of Apache Kafka
- Broker - is a node responsible for storing and processing messages. It receives messages from producers, stores them, and transmits them to consumers
- Zookeeper - is used for managing and coordinating broker nodes in the Kafka cluster. It’s responsible for tracking broker status, managing access permissions, and coordinating partition leaders
- Starting with Kafka 2.8.0, KIP-500 mode appeared - an operating mode without Zookeeper, and from Kafka 3.3+ this mode became stable and recommended. In new installations Kafka KRaft (Kafka Raft Metadata Mode) is used instead of Zookeeper
- Producer - an application or process that generates and sends messages to Kafka.
- Consumer - an application or process that receives and processes messages from Kafka.
- Topic - a theme or category where messages are stored. Producer sends messages to a specific topic, and Consumer reads messages from this topic.
- Partition - A topic can consist of multiple partitions. Each partition is an ordered immutable log of messages. Partitions allow Kafka to horizontally scale writing and reading.
- Consumer Group - is a set of consumers that together read a topic
Kafka vs Rabbit
They have different message delivery models. Kafka pulls the needed message from the topic itself - Pull.
Rabbit – implements a push model, i.e., the server pushes the message itself.
Kafka adds messages to the log, and consumers read it themselves, while
Rabbit deletes the message after processing it, and Kafka continues to store it until scheduled deletion.
Types of Message Delivery Guarantees
| Semantics | What it guarantees | Features |
| At-most-once | 0 or 1 time | Message may be lost, but no duplicates. |
| At-least-once | ≥1 time | Message is guaranteed delivered, but there may be duplicates. |
| Exactly-once | exactly 1 time | Kafka itself doesn’t implement full Exactly-Once: to achieve it, an additional mechanism is needed. For example: Outbox pattern + table on the consumer/service side to filter already processed message IDs. |
Idempotence in Kafka
Idempotence in Apache Kafka is a functionality that ensures messages won’t be processed more than once. This is important for ensuring safe and reliable data transfer between producers and consumers in the Kafka system.
Consumer Group Leader
In Apache Kafka, the consumer leader is a consumer instance responsible for reading data from the topic and coordinating message processing by other consumers. The consumer leader tracks reading progress and distributes offset information among consumers in the group.
How is the leader chosen?
- When a consumer connects to the group, the Kafka Coordinator (a special broker for the group) notifies all consumers about the group composition.
- One of the consumers is automatically chosen as leader (usually by selection algorithm - first connected or by consensus in the group).
- The leader receives current information about topic partitions that the group is subscribed to.
- The leader creates an assignment - distribution of partitions among consumers.
What does the leader do during rebalancing?
- Receives a rebalancing event from Kafka Coordinator (new consumer joins the group, consumer leaves/crashes, topics change that the group is subscribed to).
- Calculates new partition distribution among consumers (assignment).
- Sends each consumer its list of partitions.
- All consumers apply the new assignment, continue reading from the saved offset.
What options exist for offset shifting
- Message automatically shifts when sending and receiving
- Shifting occurs after processing by consumer leader
- Shifting occurs after processing by all consumers. Requires coordination (quorum) between consumers
commitSync() and commitAsync()
commitSync()
- Synchronous commit. Consumer waits for response from broker until offset is confirmed.
- Used when guaranteed offset fixation before continuing processing is required.
- Usually done after message/batch processing to avoid losing data.
commitAsync()
- Asynchronous commit. Consumer sends offset to broker but doesn’t wait for confirmation.
- Usually done after successful processing, but can also be done after reading - risk of losing the last message is minimal.
What happens if there are more/fewer partitions than consumers?
- More partitions than consumers = some consumers read multiple partitions.
- Fewer partitions than consumers = excess consumers are idle, because one partition can only belong to one consumer in the group.
- Each partition must be assigned to exactly one consumer in the group.
How is Fault Tolerance Ensured in Apache Kafka
- Partition replication - Each partition has a leader and N replicas copying data
- Reading and writing through replicas - Writing goes to the leader, then replicates to others. Reading can be served from the leader or from replicas (depending on settings).
- Quorum (ack) - Write operation is considered successful only after confirmation from the required number of replicas (
acks=all). Guarantees data consistency. - Broker mirroring (MirrorMaker) - Allows backup clusters and quick recovery after failure.
- Monitoring and auto-recovery - Quorum controller tracks broker status, automatically selects new leader on failure.
Schema Registry + Avro/Protobuf in Kafka
- Schema Registry - service that stores message schemas and manages their versioning.
- Avro/Protobuf - serialization formats that use schemas for message structure.
Classic Problems
Kafka has classic problems that almost every developer has encountered:
| Problem | What happens | Causes / Why | Short solution |
|---|---|---|---|
| Message duplication | Messages can arrive multiple times | At-least-once, consumer crash | Idempotent producer, transactions, filtering by ID |
| Consumer lag | Consumer falls behind, backlog grows | Slow processing, too few partitions | Add partitions, more consumers, processing optimization |
| Stuck partitions | Consumer hangs on partition | Blocking processing, long commit | Asynchronous processing, session timeouts, separate threads for heavy tasks |
| Hot partitions (skew) | Load is distributed unevenly | Unbalanced message keys | Use even keys, more partitions, custom partitioner |
| Rebalancing storm | Frequent group rebalances | Frequent consumer connect/disconnect, short session.timeout.ms | Increase session.timeout.ms, stable consumer instances |
Kafka Streams
Asked very rarely, but good to know)
Kafka Streams allows working with data streams from Kafka topics in real-time, applying functions like map, filter, reduce, and aggregating data without a separate cluster. Perfect for large event streams where transformations and aggregations “on the fly” are needed.
Conclusion
Today we covered key aspects and problems when working with Kafka. All this is frequently asked in interviews. The topic list is based on my experience and the experience of colleagues who went through interviews for positions from Junior to Senior.