Kafka Interview Questions and Answers for 2026

Answer: Kafka is a distributed, partitioned, replicated append-only log. Producers write records to topics, brokers persist them to disk, and consumers read them at their own pace without deleting anything for anyone else. It decouples systems that produce events from systems that react to them, at disk-speed throughput. The sentence interviewers want: it is a log with per-consumer position tracking, not a queue that deletes on read. That distinction is the root of most beginner Kafka interview questions.

Answer: A topic is a named stream of records of one kind, like orders.created or payments.settled. Logically it is a category feed; physically it is a set of partitions spread across brokers. The detail worth adding: a topic has no inherent ordering of its own, so scaling and ordering properties are partition-level, and any Kafka interview questions that follow will drill into partitions.

Answer: A partition is one append-only, immutable, totally ordered sequence of records inside a topic, with each record identified by a sequential offset. It is the unit of ordering, of parallelism, and of replication, all at once. Consequence to state: two records in different partitions have no ordering guarantee between them, which shapes how you key events.

Answer: An offset is a record’s position in its partition: a monotonically increasing integer that never repeats within that partition. Consumers use offsets as bookmarks, and committed positions are stored in an internal topic so a restarted consumer resumes where it left off. Name the storage location, __consumer_offsets, and you have answered the follow-up before it is asked.

Answer: A broker is one Kafka server: it stores partition replicas on disk and serves produce and fetch requests from clients. A cluster is a set of brokers identified by numeric IDs. Add that one broker typically hosts thousands of partition replicas and stays a thin process over the operating system page cache, and this basic item becomes one of the more informative Kafka interview questions answers you give.

Answer: A producer is the client that writes records to topics. It serializes the key and value, picks a partition by key hash or load spreading, accumulates records into batches, and sends them to the partition leader, waiting per its acks setting. The consequence interviewers probe: producer configuration, not broker hardware, decides most durability and latency outcomes.

Answer: A consumer is the client that reads records by calling poll() in a loop, receiving batches from its assigned partitions and advancing its position. Reading never removes data and never affects other consumers. The detail that earns credit: nearly all consumer behavior, fetching, heartbeating, rebalancing, is driven from that poll loop, which is why a blocked loop is the root of many intermediate Kafka interview questions.

Answer: A consumer group is a set of consumers sharing a group.id that together read a topic, with the coordinator assigning each partition to exactly one member. Within one group you get queue-like parallelism; separate groups each receive the full stream, which is publish-subscribe. That dual nature is the expected core of this entry among beginner Kafka interview questions.

Answer: The extra consumers sit idle, because a partition can only be assigned to one member of a group at a time. Adding consumers past the partition count adds cost and rebalance risk but zero throughput. The practical takeaway interviewers want: to scale a group further, you add partitions first, and that ceiling is why partition counts are chosen with growth in mind. It is a small question that reliably appears in Kafka interview questions for backend roles.

Answer: The key decides partition placement: the default partitioner hashes it, so every record with the same key lands on the same partition and keeps per-key ordering. Records with a null key are spread by the partitioner for balance and get no ordering promise. Choose keys that group events that must stay ordered, like a user ID or an order ID, and say why in one sentence.

Answer: Both, depending on how consumers organize. One consumer group acts like a competing-consumers queue because each partition goes to a single member; multiple independent groups each get the whole stream, which is publish-subscribe. That one framing, a log under both patterns, is what this classic opener among Kafka interview questions is fishing for.

Answer: The replication factor is how many copies of each partition exist across brokers. A factor of 3 survives two broker failures while staying available, which is why 3 is the production default for anything that matters. The caveat to add: copies only help if producers wait for them, which connects this answer to acks and is where most follow-up Kafka interview questions go next.

Answer: Every partition has one leader replica that serves all produces and fetches, and zero or more followers that continuously pull records from the leader to stay current. If the leader’s broker dies, an in-sync follower is promoted. All reads and writes go through the leader, which is why leadership balance across brokers matters for cluster health.

Answer: The ISR is the set of replicas fully caught up with the partition leader, meaning each has fetched everything up to the latest records within the lag window set by replica.lag.time.max.ms. Followers that fall behind are dropped from the set; caught-up ones rejoin. The ISR is the live answer to “which copies can I trust right now”, and it anchors the reliability half of Kafka interview questions.

Answer: acks=0 sends without waiting for anything, fastest and least durable. acks=1 waits for the leader to write locally, but the record can still be lost if the leader dies before followers copy it. acks=all waits for every in-sync replica, which is the durable choice for anything you cannot afford to lose. Each rung trades latency for safety, and the trade is the answer.

Answer: It is the minimum number of in-sync replicas that must acknowledge a write, checked by the broker. With replication factor 3 and min.insync.replicas=2, a write fails fast with NotEnoughReplicasException if the ISR shrinks to just the leader, instead of succeeding against one copy. Without that pairing, acks=all quietly degrades to acks=1 during incidents. That pairing appears in reliability-focused Kafka interview questions constantly, because it separates people who have run production from people who have read defaults.

Answer: It makes retries safe: the producer gets a producer ID from the broker and tags each batch with a per-partition sequence number, so the broker can detect and drop duplicates caused by retries, and ordering holds even with multiple in-flight requests. It has been enabled by default since Kafka 3.0 whenever the surrounding settings allow. Say “retries without duplicate side effects” and the follow-ups get easier.

Answer: The producer does not send each record alone; it accumulates records per partition into batches controlled by batch.size in bytes and linger.ms in waiting time, then ships the whole batch. Bigger batches amortize request overhead and compress better, at the cost of a little latency. Batching plus compression is the main reason Kafka sustains high throughput on modest hardware.

Answer: It compresses whole record batches before they leave the producer, with codecs like lz4, zstd, snappy, and gzip. Compression saves network and broker disk, since Kafka stores the batch as received, and costs CPU on both ends. zstd is the common modern pick for ratio, lz4 for speed, and the empty default none is usually leaving money on the table.

Answer: Retention bounds how long data lives, by time with retention.ms, seven days by default, or by size with retention.bytes. Deletion happens at segment granularity: a whole segment file is dropped once all its records are expired, never record by record. That is why the active segment is never deleted and why setting tiny segment sizes changes how promptly retention prunes.

Answer: Compaction, set with cleanup.policy=compact, keeps the latest record for every key and removes older records for keys that have newer values, so the topic always holds at least the current state per key. Offsets stay monotonic and unchanged. It is the right policy for changelog and state topics, and the standard contrast with time-based retention is what Kafka interview questions on this topic want to hear.

Answer: Each partition on disk is a sequence of segment files, rolled when the current one passes log.segment.bytes, one GiB by default, or log.segment.ms. Indexes map offsets to positions inside segments. Segments are the unit of retention deletion and compaction, which makes them the practical answer to “how does Kafka delete old data cheaply”.

Answer: Lag is the gap between the newest offset in a partition, the log end offset, and the group’s committed offset for that partition. Small steady lag is healthy; growing lag means consumers cannot keep up with producers. It is the single most-watched consumer health metric, and it opens more scenario Kafka interview questions than any other symptom.

Answer: A rebalance redistributes a group’s partitions among its members, triggered when a consumer joins, leaves, or is declared dead via timeouts. Processing pauses for affected partitions while ownership moves. The nuance worth one sentence: frequent rebalances are usually a symptom, typically slow processing tripping max.poll.interval.ms, not a problem in themselves, which is exactly the angle Kafka interview questions take on this topic.

Answer: It decides where a consumer starts when there is no committed offset for a partition, or the committed offset no longer exists because retention deleted it: earliest starts from the beginning, latest from the end and only new records, and none raises an error. The default is latest, which surprises people when a new group silently skips history.

Answer: With it on, the client commits offsets automatically every auto.commit.interval.ms, five seconds by default, during poll(). The risk is drift between what was committed and what was fully processed: a crash can redeliver records, or with careless ordering, silently skip them. It is fine for lossy analytics; it is wrong for anything with side effects, which is the judgment this item tests.

Answer: At-least-once retries until the broker confirms and commits offsets after processing, so records are never lost but may be delivered more than once. At-most-once commits or produces first and never retries, so nothing duplicates but failures lose records. Most business systems want at-least-once plus idempotent processing, and saying why is what elevates this answer in Kafka interview questions.

Answer: Exactly-once means each record affects downstream state exactly once, even across failures and retries. In Kafka it is assembled from idempotent producers, transactions that atomically publish output records and commit input offsets, and consumers reading with read_committed so aborted writes are skipped. It is end-to-end processing semantics, not a network magic trick, and scoping that claim correctly is the whole answer.

Answer: It is a separate service that stores schemas and enforces compatibility rules as they evolve. Producers serialize payloads with a registered schema and send its ID alongside compact binary data; consumers fetch the schema by ID and deserialize. The point is contract management: a producer cannot ship a breaking change, because the registry rejects it. Kafka interview questions on data quality almost always detour here.

Answer: Avro: compact binary, mature compatibility tooling, the long-time ecosystem default. Protobuf: strong typing and first-class code generation across many languages, common in gRPC-heavy shops. JSON: human-readable and dependency-free but larger and weakly typed, fine for low-volume topics or debugging feeds. The expected framing is tradeoffs, not a favorite: schema rigor, payload size, and tooling fit decide it.

Answer: Before KRaft, ZooKeeper stored cluster metadata: broker registration, config, ACLs, and it ran controller elections. That meant two distributed systems to operate, secure, and monitor, with metadata synchronized between them. Legacy fleets still run it, so the question survives in interviews, usually phrased as “have you migrated off it and why”.

Answer: KRaft is Kafka’s built-in metadata protocol: a quorum of controllers keeps cluster metadata in an internal Raft-replicated log, replacing ZooKeeper entirely. Operations get simpler, controller failover gets faster, and the metadata system scales to millions of partitions. It is the only mode since Kafka 4.0, which makes KRaft awareness one of the current-events items in modern Kafka interview questions.

Answer: kafka-topics.sh --bootstrap-server broker:9092 --list enumerates topics, and --describe --topic name shows the working detail: per-partition leaders, replicas, ISR membership, and config overrides. Reading that output aloud, explaining leader and ISR columns, is the fastest credibility signal in a terminal round, and it is why CLI fluency shows up in Kafka interview questions for every ops-adjacent role.

Answer: kafka-console-producer.sh --bootstrap-server broker:9092 --topic t writes lines as records, with --property parse.key=true --property key.separator=: to send keyed records. kafka-console-consumer.sh --topic t --from-beginning --group smoke-test reads them back, and adding --property print.key=true shows keys. It is the smoke test of choice for auth, connectivity, and serialization checks before any application deploy.

Answer: Kafka keeps its JVM heap small and lets the operating system cache log segments, because produce is sequential append and the hottest reads are the newest records, which are exactly what the cache holds. The payoff: garbage collection stays cheap and memory scales with data, not heap settings. Consequences interviewers like: never drop caches on a busy broker, and never let a broker swap.

Answer: A tombstone is a record with a null value written to a compacted topic, marking that a key was deleted. It survives compaction for a grace period, delete.retention.ms, 24 hours by default, so consumers have time to see the deletion, then it is removed. Without tombstones, deleted keys would resurrect from old records, which is the failure mode this item exists to check.

Answer: It replicates topics and consumer group offsets between Kafka clusters, for disaster recovery, cloud migration, or aggregation into a central cluster. It runs on Connect under the hood, translates offsets as it copies, and records checkpoints so failover can resume consumers near where they left off. Knowing that mirrored topics get a source prefix by default is a detail that surfaces in cross-region Kafka interview questions.

Answer: A DLQ is a pattern, not a built-in: after fixed retries, a record that keeps failing processing is written to a companion topic like orders.created-dlq for inspection, and the pipeline moves on. Kafka Streams and Connect ship handlers that do this for deserialization and sink failures; plain consumers roll it themselves. Naming the poison-pill risk, one bad record stalling the whole partition, is what rounds the answer out.

Answer: Three cases. An explicit partition in the record wins. Otherwise, a record with a key is mapped by the default partitioner, murmur2 hash of the key modulo partition count, so keys distribute stably. A keyless record uses the sticky partitioner: the client fills one partition’s batch before switching, which keeps batches full and throughput high without the request overhead of strict round-robin. The caveat worth stating: partition counts changing reshuffles that hash mapping, which breaks per-key ordering assumptions.

Answer: send() serializes key and value, runs the partitioner, and appends the record to an in-memory per-partition batch inside the accumulator, bounded by buffer.memory. A background sender thread drains ready batches, compresses them, and ships them to the partition leader, honoring linger.ms and batch.size, then tracks the in-flight request until acks are satisfied or delivery.timeout.ms expires. Saying that the future returned by send() is where errors surface, and that blocking on it serializes your app, is the detail senior panels listen for. Narrating that path cleanly is a signature moment in client-side Kafka interview questions.

Answer: At InitProducerId, the broker assigns the producer a producer ID, and the producer tags every batch per partition with a monotonically increasing sequence number. The broker tracks the last sequence per producer-partition: duplicates from retried sends are dropped, and out-of-order batches are rejected. Because brokers cache a small window of sequence state, idempotence holds with up to five in-flight requests, which is exactly why max.in.flight.requests.per.connection must stay at or below 5 with idempotence on.

Answer: Retries without idempotence can reorder records: batch 1 fails, batch 2 succeeds, batch 1 retries and lands late. With enable.idempotence=true, sequence numbers let the broker enforce order even across retries, so max.in.flight.requests.per.connection can stay as high as 5 without reordering. Without idempotence, strict ordering forces either a single in-flight request or accepting the risk. This triangle, retries, in-flight count, ordering, is a standard probe in intermediate Kafka interview questions.

Answer: acks=all for broker-side confirmation, retries left high since failures retry internally, and delivery.timeout.ms as the overall budget that bounds retries plus linger plus broker time. Pair the topic’s min.insync.replicas at 2 or more with replication factor 3. Then decide policy per failure: send() futures must be checked, and TransactionTimeout or OutOfOrderSequence errors mean the producer instance must be recreated, not reused. A checklist delivered in that order is what the question is after.

Answer: A transactional producer sets transactional.id; on init it registers with the transaction coordinator and bumps an epoch, which fences any zombie instance of the same ID. It can then atomically publish records to many partitions and, via sendOffsetsToTransaction, commit consumed offsets in the same transaction. Commit writes control records that mark the batch committed or aborted. The use case is exactly-once consume-transform-produce flows, not generic database-style transactions.

Answer: A read_committed consumer reads only up to the last stable offset, the LSO, the point before any open transaction, and skips records from aborted transactions using the control markers. The operational consequence: a hung open transaction pins the LSO and stalls those consumers until the transaction times out and aborts. read_uncommitted reads everything immediately, aborted records included. Naming the LSO mechanism is what this question exists to surface.

Answer: For every group, one broker is the GroupCoordinator, chosen by hashing the group ID onto the 50 partitions of __consumer_offsets. Members send heartbeats to it, it runs the join and sync handshake on membership changes, and it persists group state, including generation IDs, in that compacted topic. Knowing that coordinator selection is deterministic from the group name explains why moving a group’s load around is a routing question, and it is a favored detail in intermediate Kafka interview questions.

Answer: In __consumer_offsets, an internal, compacted, 50-partition topic keyed by group, topic, and partition. commitSync sends the current positions and blocks until confirmed; commitAsync fires without blocking and retries in the background, at the cost of possibly committing a stale position if a later one already succeeded. Offsets are just records there, which is why they survive consumer restarts and why replay is simply resetting them. Both facts are favorite pivots in Kafka interview questions about recovery.

Answer: commitSync gives certainty at the cost of latency in the poll loop; commitAsync keeps the loop tight but can commit stale offsets after a rebalance. The production pattern combines them: async commits for routine progress, sync on shutdown and on rebalance callbacks, so you neither stall throughput nor lose position at the moments that matter. Stating that combination, and when each half fires, is the complete answer.

Answer: Every member stops its fetches and revokes all owned partitions, then sends JoinGroup. The coordinator picks a leader member, waits for the full membership, and returns the roster; the leader computes the assignment and sends it via SyncGroup, which distributes it to everyone. Members then fetch from their newly owned partitions. During that window the group processes nothing, which is why rebalance frequency is a latency line item, and why the diagram below is worth rehearsing for Kafka interview questions on this topic.

Answer: The protocol becomes incremental: instead of revoking everything up front, members keep ownership and only release partitions that must move, in one or two balanced rounds. Rebalance callbacks hand you the specific revoked set so you can commit exactly those offsets. The catch: all members of the group must support the cooperative protocol, and rolling an eager group to cooperative needs the double-hop upgrade path, since mixing strategies aborts the rebalance with an inconsistent protocol error.

Answer: session.timeout.ms is broker-side liveness: how long the coordinator waits for a heartbeat before declaring the member dead and rebalancing. max.poll.interval.ms is client-side work health: how long may pass between poll() calls before the client concludes it is stuck, leaves the group proactively, and triggers the rebalance itself. The first catches crashed processes; the second catches wedged processing. Misdiagnosing which one fired is the most common failure in rebalance-focused Kafka interview questions.

Answer: Setting group.instance.id makes a member a static member: on restart within the session timeout it rejoins under the same identity and reclaim its old partitions without triggering a rebalance. It exists for stateful consumers and rolling restarts, where a re-joining dynamic member would otherwise shuffle partitions needlessly. The caveat: identity collisions, two processes with the same instance ID, fence each other off, so the ID must be stable per deployment slot, not per host.

Answer: RangeAssignor, the legacy default, assigns per-topic in sorted blocks, which can stack multiple topics’ first partitions on one consumer. RoundRobinAssignor spreads partitions evenly across members topic by topic. StickyAssignor balances while minimizing movement between rebalances. CooperativeStickyAssignor adds incremental revocation, so only moved partitions pause. The modern default for new groups is cooperative-sticky, and being able to say why, fewer paused partitions per rebalance, answers the follow-up in the same breath.

Answer: poll() issues fetch requests to partition leaders for the consumer’s assignments; the broker returns as soon as fetch.min.bytes accumulate or fetch.max.wait.ms elapses, whichever first, up to max.partition.fetch.bytes per partition. The client deserializes, hands records to your handler, and updates position. Raising fetch.min.bytes improves throughput at the cost of broker-side waiting, and decompression happens client-side, so CPU per consumer matters when producers compress aggressively.

Answer: The log end offset, LEO, is the next offset a partition will assign, the tip of the log including unreplicated writes. The high watermark is the highest offset replicated to every in-sync replica, and consumers can only read up to it, which is why freshly produced records are not instantly visible. After a leader change, leader-epoch checkpoints let followers and consumers detect truncation and avoid reading diverged data. This pair is the mechanism behind lag numbers, durability claims, and visibility delays all at once.

Answer: The leader tracks how long since each follower last fetched up to the log tip; exceeding replica.lag.time.max.ms, 30 seconds by default, evicts it from the ISR, and a caught-up follower is added back. Under-replicated partitions mean ISR count is below replication factor. The operational points: sustained ISR shrinkage usually means network or disk trouble rather than config, and a shrunken ISR with acks=all reduces the fault tolerance your writes actually enjoy.

Answer: With unclean.leader.election.enable=false, the default, a partition whose last in-sync replicas are all dead stays offline rather than promote an out-of-sync follower, choosing availability-of-consistency over availability-of-service. Enabling it lets a stale replica lead, recovering the partition at the cost of losing the records it never received. You would enable it, deliberately and usually per-topic, only for data where losing recent records beats staying offline, like high-volume metrics. That trade articulation is the entire interview point.

Answer: The controller is the broker, or under KRaft the active controller node, responsible for cluster-wide decisions: electing partition leaders after broker failures, propagating metadata updates, and managing partition reassignments. With ZooKeeper it watched a ZK path and pushed state to brokers, a slower two-hop design. KRaft controllers run a Raft quorum over the metadata log, so failover is fast and metadata throughput scales to millions of partitions. Knowing which entity you are debugging, controller versus broker, shapes every cluster-level investigation.

Answer: The active segment rolls closed once it passes log.segment.bytes or log.segment.ms, and a background checker running every log.retention.check.interval.ms marks fully expired segments for deletion. Deletion is a file unlink, never a record rewrite, so it is cheap and does not stall traffic. The subtle part interviewers probe: retention clock granularity is the segment, so a huge segment with few old records cannot shrink until it rolls, and the active segment is always exempt.

Answer: Log cleaner threads split a partition into a clean tail, already compacted, and a dirty head, build an offset map of the latest offset per key from the dirty section, then rewrite segments, copying only the latest surviving record per key. Offsets are preserved, gaps and all. Tombstones linger for delete.retention.ms so deletions propagate, and min.cleanable.dirty.ratio controls how much dirt justifies a pass. Cleaner throughput is throttled, which is why compaction can lag behind on hot topics without any error being raised. That silent-stall behavior is a classic trap in Kafka interview questions about compaction.

Answer: Four designs stack. Writes are sequential appends, which disks love, instead of random updates. Reads of recent data come from the OS page cache, so the broker avoids copying through the JVM heap. The consumer path uses zero-copy transfer, sendfile, moving bytes from page cache straight to the socket without user-space copies. And client-side batching plus compression amortize every request. Name all four and note they compound; the compounding is the real answer, and it is why hardware-sizing Kafka interview questions start from throughput per broker, not CPU count.

Answer: num.network.threads handle request transfer, num.io.threads handle disk work, and num.replica.fetchers decide how fast followers replicate. socket.request.max.bytes bounds request size, and per-broker replica.fetch.max.bytes must exceed your largest record batch or replication stalls. The method: watch RequestHandlerAvgIdlePercent; network threads idling low means clients wait in the processor queue, io threads idling low means disk or fetchers are the constraint. Tuning without reading those two is guessing.

Answer: Raise batch.size, often to 64 or 128 KiB, give linger.ms 5 to 20 so batches fill, enable compression.type=zstd or lz4, and size buffer.memory for the burst rate so sends do not block on the accumulator. Keep max.in.flight at 5 with idempotence on and acks=all, which costs little at these batch sizes. Then validate with kafka-producer-perf-test.sh against the real topic, because the interplay of batch and compression is data-dependent.

Answer: linger.ms=0 so sends leave immediately, modest batches as a safety net for bursts, acks=1 only if the durability tradeoff is genuinely acceptable, and consumer-side fetch.min.bytes=1 so brokers answer instantly instead of waiting to fill. Monitor the p99, not the average, because batching latency is a tail phenomenon. The honest framing panels reward: latency tuning spends the throughput budget you just built, so every setting is a placement on that curve, not a free win.

Answer: Broker-enforced limits per user or client ID: producer and consumer byte rates, and request processing time share. Set them with kafka-configs.sh --alter --add-config 'producer_byte_rate=52428800,consumer_byte_rate=104857600' --entity-type clients --entity-name app1. They exist so one runaway batch job cannot starve interactive traffic, and throttled clients see responses delayed, not errors. Quotas are the standard answer to “how do you keep one bad tenant from hurting others” in shared-cluster Kafka interview questions.

Answer: Calling consumer.pause() on slow partitions keeps the poll loop alive, so heartbeats flow and the member stays in the group, while you drain work to a downstream that is behind; resume() re-enables fetching when capacity returns. Sleeping inside the loop risks max.poll.interval.ms, triggering a rebalance storm exactly when the system is already strained. Pause-based backpressure is the professional answer, and naming the failure it avoids is what earns the point.

Answer: seek(partition, offset) sets the position directly; seekToBeginning and seekToEnd do the obvious. For time-based replay, offsetsForTimes maps a timestamp to the first offset at or after it, then seek there, which is the standard incident-replay move. Bulk operations belong on the CLI: kafka-consumer-groups.sh --reset-offsets --group g --topic t --to-datetime 2026-01-01T00:00:00.000 --execute, with a --dry-run first. Stop the consumers or expect the running poll loop to fight the reset.

Answer: Fifty partitions by default via offsets.topic.num.partitions, compacted so only the latest position per group-topic-partition key survives, with replication factor governed by offsets.topic.replication.factor. A group’s coordinator is whichever broker owns the partition its name hashes to. It is a normal Kafka topic under the hood, which means it can lag, corrupt, or get under-replicated like any other, and treating it as a normal topic is exactly the insight intermediate Kafka interview questions want there.

Answer: At-most-once: commit before processing, or produce with acks=0 and no retries; fast, lossy on failure. At-least-once: acks=all, retries on, commit after processing completes; duplicates possible, so consumers need idempotent writes keyed by record key or event ID. Exactly-once: transactional producer with a stable transactional.id, offsets committed through sendOffsetsToTransaction, consumers on read_committed, or Kafka Streams with exactly_once_v2. Reciting the config triple per semantic is the pass bar for this family of Kafka interview questions.

Answer: When the work is stateful: joins between streams, windowed aggregations, deduplication, or any transformation that needs local state with fault tolerance. Streams gives you the DSL, state stores backed by changelog topics, exactly-once processing, and elastic rebalancing of tasks, all without an external cluster. For stateless forwarding or simple per-record side effects, a plain consumer is fewer moving parts. Choosing by statefulness, not fashion, is the judgment this question grades.

Answer: A KStream is an unbounded sequence of records where every entry is an event, so two records for one key are two facts. A KTable is a changelog view where each record is an upsert, so only the latest value per key is table state, and old values are history. The classic example: two clicks are two KStream records but one KTable row per user. Joins come in stream-table, table-table, and windowed stream-stream forms, and picking the wrong abstraction produces subtly wrong aggregates, which is precisely the bug this question is screening for.

Answer: In three internal topics maintained by the worker cluster: connect-configs, compacted connector and task configuration, connect-offsets, source positions per partition so a restarted connector resumes rather than re-extracts, and connect-status, the current state of every connector and task. Workers coordinate through these topics, which is what makes Connect distributed-mode resilient. Because offsets live in Kafka, resetting a source connector is a matter of manipulating those records, and saying so demonstrates real operational depth.

Answer: The registry checks each new schema version against the subject’s policy. Backward, the default: new consumers can read data written with the old schema, which permits adding fields with defaults and removing optional ones. Forward: old consumers read new data. Full: both directions, the strictest practical setting, with transitive variants checking against all history rather than the last version. Violations are rejected at registration time, which is the whole point: incompatible payloads never reach a topic.

Answer: Three layers. Authentication: SASL mechanisms like SCRAM-SHA-512 or mTLS certificates prove who the client is. Encryption: TLS on the wire, and often encrypted disks. Authorization: ACLs grant operations, like Read on a topic or Read on a group, to principals, evaluated by the authorizer. The interview expectation is not policy authorship; it is knowing that authn and authz are separate checks, that listeners can mix TLS and SASL, and that a “connection refused” with a security misconfiguration usually appears as a timeout or an authentication error, not a clean refusal.

Answer: With broker.rack set, the replica placer spreads each partition’s replicas across racks, so losing one rack or availability zone leaves every partition with surviving replicas. It shapes placement only, not election: leadership still prefers in-sync replicas by normal rules. Client-side, a rack-aware replica selector lets consumers read from the nearest replica instead of always the leader, cutting cross-zone traffic cost. Naming that read-path option is the detail that separates configured clusters from defaulted ones.

Answer: A broker feature that uploads closed log segments to cheap object storage like S3 while keeping a small local window defined by local.retention.ms, decoupling retention from broker disk capacity. Consumers read hot segments locally and cold ones remotely, transparently. It changes capacity planning math: long retention stops meaning huge local disks. The current caveats to mention: read latency for cold data, and the operational maturity of specific implementations, which is why panels ask it as a tradeoff question and as one of the newer entries in platform-focused Kafka interview questions.

Answer: It wraps each Streams task’s work in a transaction: consumed input offsets, produced output records, and state-store changelog updates commit atomically, and the consumer side reads its inputs with committed visibility, fencing zombies through producer epochs. Compared to the v1 design it no longer bridges through a separate coordination step, so it is simpler and faster. State that the cost is throughput and that the guarantee covers Kafka-to-Kafka processing only; the moment you write to a database, you are back to designing idempotency yourself.

Answer: The classic protocol made the client-side group leader compute assignments through JoinGroup and SyncGroup handshakes, so rebalances were slow and membership scales poorly. KIP-848 moves assignment server-side: brokers manage membership incrementally, members add or leave without full-group stop-the-world rounds, and heartbeats ride a lightweight path. It is the default consumer protocol as of Kafka 4.0. The interview-grade summary: rebalance cost drops from whole-group to only-affected-partitions, at the broker, which is where most rebalance pain lived.

Answer: First, split the problem space: kafka-consumer-groups.sh --describe --group g to see lag per partition and per member. If one member holds all the lag, suspect assignment skew or a hot partition; if lag is spread evenly, consumers as a class got slower. Second, correlate with the deploy: new consumer code, changed max.poll.records, added per-record work, or a slower downstream dependency. Third, check the broker side for changed conditions: latency on fetch requests, disk waits. The decision point: growing end offset means producer volume, stalled committed offset means consumer health. That split, broker versus client versus code, is the entire frame for lag-crisis Kafka interview questions.

Answer: Read the group’s recent history first: coordinator logs and kafka-consumer-groups.sh --describe --group g --state for the current generation and members. Then split causes. Poll-side: processing time exceeding max.poll.interval.ms, visible as consumers leaving voluntarily with that error, fixed by lowering max.poll.records or raising the interval. Liveness-side: flaky heartbeats from GC pauses or unstable networks, visible as session timeouts, fixed by GC tuning or saner session.timeout.ms. Deployment-side: overlapping instances fighting over one group.id or colliding static IDs. The order matters, work-side before liveness-side, because slow processing is by far the most common culprit, and saying so up front is what scenario Kafka interview questions reward.

Answer: Accept the premise first: with at-least-once semantics, redelivery after a crash is expected behavior, not a bug, because offsets commit after processing by design. The fix belongs in the effect, not the transport. Make processing idempotent: deduplicate on the record key or a business event ID before applying side effects, write with upserts keyed by event ID, or track processed IDs in a store. If duplicates are truly unacceptable, move the pipeline to transactions with offsets committed atomically to outputs. Saying “you cannot configure away at-least-once duplicates, you design them away” is the sentence this scenario Kafka interview questions item exists to hear.

Answer: Confirm with per-partition rates before acting, then diagnose the key distribution: a hot key, like one huge tenant on a tenant-ID key, is the usual cause, and repartitioning to more partitions cannot help because the hash still sends that key to one partition. Short term, scale the downstream for that partition’s consumer or isolate the tenant to its own topic. Long term, change the keying: compound keys like tenant plus shard spread the heat, at the cost of cross-partition ordering for that tenant. A custom partitioner is the last resort because every consumer and producer must adopt it. Stating “more partitions do not fix a hot key” early is the differentiator in skew Kafka interview questions.

Answer: First, stop the bleeding risk: check which log directories are filling with kafka-log-dirs.sh --describe, and confirm free space alerts match. Second, find the growth source: an unexpectedly huge topic, a topic whose retention.ms was dropped via a bad config push, or a partition whose traffic spiked. Third, buy time safely: lower retention on the fattest non-critical topics with a dynamic config via kafka-configs.sh --alter --add-config retention.ms=3600000, which deletes whole segments at the next retention pass; do not delete log files by hand, that corrupts the broker. Fourth, plan the durable fix: capacity, tiered storage, or moving partitions with a reassignment. The no-manual-deletion rule, stated firmly, is what this scenario Kafka interview questions classic is really testing.

Answer: Kafka to Kafka is the case where exactly-once is real. Use a transactional producer with a fixed transactional.id per pipeline instance; on start it fences any zombie from a previous epoch. In the loop: begin the transaction, poll records, process, produce outputs, commit input offsets with sendOffsetsToTransaction, then commit, so outputs and offsets publish atomically. Consumers downstream read with isolation.level=read_committed to skip aborted batches. Set transaction.timeout.ms longer than your worst processing batch, and keep batches small enough that a timed-out transaction does not pin the LSO for everyone. On Kafka Streams, the same design collapses to processing.guarantee=exactly_once_v2. Explaining the fence, the atomic commit, and the reader contract is the full-credit shape for design Kafka interview questions like this one.

Answer: That error means poll calls are landing mid-rebalance, and if it repeats, the group is trapped in a rebalance loop. My order: first, check whether processing time per poll exceeds max.poll.interval.ms, because each timeout ejects the member, triggers a rebalance, and the remaining members inherit the backlog, which makes them slower and next to time out: a cascade. Second, look for a member crashing and rejoining repeatedly, deployment flapping or OOM kills. Third, verify the coordinator is stable; a coordinator moving due to broker churn also churns the group. Remedies in order: cut max.poll.records, move heavy work off the poll thread with pause and resume backpressure, then raise the interval. Narrating the cascade mechanism is the core of this scenario Kafka interview questions entry.

Answer: This is the inverse duplicate: someone committed before processing finished, and a crash in between dropped records permanently. Diagnostic order: confirm the gap, compare the earliest unprocessed offsets against the downstream’s max event ID or timestamp. Then reconstruct the commit order in the code: auto-commit enabled plus slow processing commits positions for records still in memory, or a commit call placed before the write call. Fix by inverting the order: process, then commit, restoring at-least-once semantics, and accept the duplicates that implies by making the sink idempotent. The lesson to say out loud: commit order is the entire difference between at-least-once and at-most-once, and this failure is the at-most-once accident.

Answer: “Expiring records” means batches sat in the accumulator past delivery.timeout.ms without being confirmed. Order of checks: first, broker-side health, produce request latency and request handler idle percent on the target brokers, because a saturated cluster is the most common cause. Second, buffer pressure: buffer.memory exhaustion blocks sends and ages batches; the client metric for record queue time tells you. Third, config mismatch: linger.ms plus retries plus slow acks exceeding the timeout budget. Fourth, an ISR that keeps shrinking with acks=all, making every write wait and occasionally fail. Remedies follow the finding: capacity, bigger buffer, longer timeout, or fixing replication. That ordered list is what timeout-flavored Kafka interview questions want spoken back.

Answer: Clients hold cached metadata naming the old leader; until they refresh, sends and fetches hit a broker that no longer leads those partitions, and the broker answers with that error. Usually it self-heals as metadata refreshes. Persistent errors mean something else: preferred leader election did not run, so leadership is lopsided, checked with kafka-topics.sh --describe comparing leader versus preferred replica; or the restarted broker cannot catch up and is not rejoining ISRs, visible as under-replicated partitions. Fixes: trigger kafka-leader-election.sh --election-type preferred for stuck partitions, and investigate the catch-up failure, usually disk or network. Distinguishing the transient refresh case from the stuck-election case is the graded judgment.

Answer: Identify scope first: kafka-topics.sh --describe --under-replicated-partitions, clustered on one broker means that broker is sick; spread across the cluster means a systemic cause. Then per broker: is it fetching, check replication throughput and whether the broker fell out of ISR entirely; disk latency and utilization on its log dirs; network errors to its peers; and whether a recent config or partition count change multiplied its load. A dead disk on one log dir needs the log dir failure path, moving replicas off with reassignment. Do not restart blindly: a restart of a broker that cannot replicate just adds another leader election wave. Scope before action is the whole answer.

Answer: Before, find why, or the restart dies the same way: dmesg for the kill record, heap settings, and whether the process was actually Kafka or a colocated process; broker heaps are small by design, so a genuine Kafka heap OOM points at oversized requests, connection floods, or a misconfigured heap. Check connection counts and socket.request.max.bytes abuse. After restart: watch ISR recovery, confirm leaders rebalance back rather than piling elsewhere, check that replication catch-up does not saturate disks, and verify consumer lag drains. The interview point: brokers are restart-tolerant, but an unexplained OOM is a recurring outage, so root cause first is the professional order for this scenario Kafka interview questions staple.

Answer: Read one client’s error precisely first, because the error class routes you: TimeoutException on a TLS listener usually means a plaintext client hitting a TLS port or a missing truststore; SaslAuthenticationException means credentials or mechanism mismatch; TopicAuthorizationException means auth worked but ACLs are missing, a different fix entirely. Then diff the change: listener security protocol map, SASL mechanism lists, JAAS config, and client properties on one known-good and one failing client. Fix forward on the client side or roll back the listener change, since a broker that cannot authenticate anything is a full outage. Separating authentication errors from authorization errors in the first ten seconds is the entire skill this scenario probes.

Answer: Name the breakage first: the key-to-partition mapping is hash modulo partition count, so after the change almost every existing key maps elsewhere, breaking per-key ordering across the transition and invalidating any state that assumed partition locality. If that is unacceptable, the safe path is a new topic with 24 partitions, dual-write or mirror during migration, then cut consumers over and retire the old one. If ordering tolerance exists, alter the topic, expect a rebalance, and let key distribution settle. Either way, capacity-check the cluster: 12 more partition replicas mean more files, more fetchers, more leader slots. Leading with the ordering consequence is what this Kafka interview questions classic is really asking for.

Answer: Order of suspects: dynamic config overrides first, because a bad automation push is the usual culprit, checked with kafka-configs.sh --describe --entity-type topics --entity-name t, where a topic-level retention.ms or retention.bytes beats broker defaults silently. Second, size-based retention: if retention.bytes or the broker default is set and traffic spiked, deletion is by size, not time. Third, segment geometry: tiny segment.bytes makes deletion granularity aggressive, since only whole segments expire but small ones expire sooner. Fourth, log compaction misunderstanding: on compacted topics old values vanish by design. Fifth, human action: someone reset offsets and purged, or recreated the topic. Config overrides before exotic causes is the ordering discipline this item rewards.

Answer: Check what compaction can and cannot do. First, key cardinality: compaction keeps one record per key, so ten million unique keys is ten million records forever; growth from unique keys is not a compaction failure. Second, cleaner health: log.cleaner.enable must be true, and cleaner thread metrics or logs show passes and dirty ratios; a crashed cleaner thread silently stops compaction. Third, the active segment never compacts, so a huge segment.bytes delays reductions. Fourth, min.cleanable.dirty.ratio and min.compaction.lag.ms gate passes, so low-churn topics compact rarely. Fifth, tombstone accumulation with a long delete.retention.ms. The reframe to deliver: compaction bounds state per key, not total size, and that sentence resolves most compaction Kafka interview questions.

Answer: Work the constraints in order. Consumer count: if the group still has fewer members than partitions, the bottleneck is members, not partitions, so scale the deployment to the new partition count. Key skew: if traffic concentrates on few keys, extra partitions stay empty and nothing improves. Downstream: if the sink database is the wall, more consumers just queue harder at the insert; measure sink latency per record. Client config: fetch sizes and processing model may cap per-member throughput regardless of partitions. Leader locality: new replicas may concentrate leadership on one broker. Confirming which constraint moved the number, by changing one thing at a time, is exactly what scenario Kafka interview questions about scaling are grading.

Answer: Treat it as a capacity shape problem, not an incident. Confirm the pattern correlates with an upstream batch: producers flooding the topic at 02:00 while steady-state consumer capacity only matches steady-state volume. Then choose deliberately: smooth the producer side with pacing or splitting the batch job, which is usually cheapest; or size consumers for peak, more partitions so more members can join, plus throughput tuning on the consumer; or isolate the batch onto its own topic and consumer group so the interactive stream stays flat. Add lead measurement: alert on lag slope, not absolute lag, so you see the drain rate improving after the fix. Naming the three levers, producer shape, consumer capacity, isolation, is the complete answer.

Answer: First, measure precisely: checkpoint topics and MM2 metrics give per-topic lag in records and offsets. Then find the constraint: the MM2 Connect workers’ throughput, usually limited by fetch settings and task count; source or target broker saturation; or WAN bandwidth, the classic ceiling for large histories. Check whether lag is uniform or concentrated in high-volume topics. For the failover question: consumers resume from translated offsets in checkpoints, so cost equals current lag, plus anything the source writes between decision and cutover, which is why DR runs regularly rehearse the cutover with sync.group.offsets enabled. Being able to quantify failover cost from lag is the senior layer of cross-cluster Kafka interview questions.

Answer: Order the levers. First, backpressure without leaving the group: pause() the partitions, let the database drain, resume() when latency recovers; heartbeats continue because the loop keeps polling. Second, decouple with an internal buffer: move slow writes to a worker pool bounded in size, committing offsets only for fully completed work, so a crash redelivers rather than loses. Third, batch the sink: one transactional upsert of a whole poll beats per-record writes. Fourth, negotiate capacity honestly: if the database cannot sustain average input rate, no consumer trick fixes it. The graded judgment: never sleep inside the poll loop past its interval, because that trades a slow consumer for a rebalancing one.

Answer: Heap fine is the clue: Kafka wants page cache, not heap, so check memory pressure next. If the OS is evicting cache or the box swapped, every read of recent segments goes to disk and produce acks with acks=all slow down behind replication reads; check swap, cache hit behavior, and vm.swappiness. Then look at request queues: network processor and request handler idle percentages, because thread saturation shows as latency with idle disks. Then disk: await and utilization on the log volumes, and whether leadership concentrates on one broker at peak. Finally, replication load: under-replicated stretches mean follower fetches compete with clients. Layered in that order, memory, queues, disks, leadership, replication, is the diagnostic spine this scenario Kafka interview questions item teaches.

Answer: The mechanism: the producer sent batch 1, timed out or got an error, sent batch 2 which succeeded, then retried batch 1, which the broker accepted late, because without idempotence the broker has no sequence information to reject it. Confirm by checking enable.idempotence and max.in.flight.requests.per.connection on the offending producer. Prevention: idempotence on keeps order across retries with up to five in-flight batches; with very old clients, either drop in-flight to 1 or handle reordering downstream by event timestamp. Also verify keys: if ordering matters per entity, the entity must be the key, or no producer setting can save you. Mechanism then config then keying is the full-credit order.

Answer: The consumer reads up to the last stable offset, and an open transaction pins it: nothing beyond the unfinished transaction is visible, so the partition looks stalled even though records flow. The system self-heals: the transaction coordinator aborts the transaction when transaction.timeout.ms expires, writes an abort marker, and the LSO advances past it. Diagnosis confirms the shape: transaction markers in the log via kafka-dump-log.sh --print-data-log, and a stalled LSO on the affected partition. Prevention: producer transaction timeouts sized well under the consumer-visible delay you can tolerate, producers that abort cleanly on failure, and small transactions. Naming the LSO as the pinning mechanism is the point of this scenario.

Answer: Split the likely causes in order. Protocol mismatch: the new version changed the assignment strategy or group protocol, and mixing members fails the join with an inconsistent protocol error, so each side retries, which looks like flapping; fix by draining one version fully before the other joins, or aligning the strategy config. Static membership collisions: both versions shipping the same group.instance.id fence each other repeatedly. Config drift: different session.timeout.ms values outside broker bounds reject one side’s membership. Group identity: an accidental group.id change creates a second group that starves while the first stalls. The general rule to state: consumer groups require protocol homogeneity, and rolling upgrades must respect that, which is the exact insight this scenario Kafka interview questions item targets.

Answer: Lead with the four that move first. UnderReplicatedPartitions rising means replication health is degrading before anything is lost. RequestHandlerAvgIdlePercent and network processor idle falling toward zero mean request saturation before latency alerts fire. Log dir disk usage and await trending up predict the disk-full and slow-segment events that cause the worst incidents. ActiveControllerCount must be exactly one per cluster: zero or two means an election problem, always urgent. Around those, watch ISR shrink and expand rates, produce and fetch latency percentiles, connection counts, and per-group consumer lag slope. The framing that lands in interviews: alerts on rates and slack, not absolutes, because a healthy Kafka cluster announces its problems in trends long before partitions go offline.

Answer: Understand the mechanics first: with unclean.leader.election.enable=false, only an in-sync replica may lead. If every ISR member is down, or their log dirs failed, the partition stays leaderless by design, protecting data consistency at the cost of availability. Diagnose: kafka-topics.sh --describe shows Leader: none and the replica list; compare against broker health and log dir state on the replicas. Options in order: bring an ISR member back, which is the clean path; restore its disk or data if corrupted; or, as a deliberate, documented decision for loss-tolerant data, temporarily enable unclean election for that topic and accept the gap between ISR progress and the old leader’s tip. The interview scoring: stating that the leaderlessness is a consistency guard, not a bug, before proposing any fix.