Distributed Systems Interview Questions for 2026

Answer: A set of independent machines that cooperate over a network to appear, to users, like one coherent system. No shared memory and no shared clock, so every machine’s view of the world can briefly disagree with the others. That disagreement is the root of almost every distributed systems interview question, from consistency to clocks.

Answer: A single box hits hard ceilings: CPU sockets, RAM addressability, disk throughput, and above all a blast radius of one. Distributing buys fault tolerance, horizontal scale, and geographic proximity to users. The trade is that you inherit partial failure, any component can die while the rest keeps running, which is precisely the situation single-machine code never had to handle. Partial failure is also the lens most distributed systems interview questions are viewed through.

Answer: During a network partition, a distributed system must choose between consistency, every read sees the latest write, and availability, every request gets an answer. Partition tolerance is not optional, so the choice is really C versus A while the network is broken. It says nothing about normal operation, which is exactly the gap PACELC fills and where follow-up distributed systems interview questions go.

Answer: The cluster splits into groups that cannot talk to each other, even though every node may still be up and healthy. It can be a dead switch, a security group change, or a congested link dropping packets. The danger is not silence, it is that both sides keep working and can produce conflicting state, which is why partitions dominate distributed systems interview questions about failure.

Answer: Keeping copies of the same data on multiple machines, for durability when one dies and for read scale when one is too slow. The cost is keeping those copies agree: the instant a write lands on some replicas but not others, reads can disagree, and consistency becomes a design decision rather than a free property.

Answer: Partitioning, also called sharding, splits the dataset so each node owns a slice, while replication copies the same slice onto several nodes. They compose: each shard is itself replicated for safety. Partitioning solves dataset and write scale; replication solves durability and read scale. Confusing the two is a classic early stumble in distributed systems interview questions.

Answer: One leader accepts writes, records them in a log, and ships them to followers that apply the changes and serve reads. If the leader fails, a follower is promoted. It gives a single ordering for writes, which keeps reasoning simple, at the price of a failover window and, with asynchronous shipping, the chance the promoted follower is missing the last few writes. That failover caveat is the follow-up most distributed systems interview questions attach here.

Answer: A model where, if writes stop, every replica converges to the same value eventually, but until then reads can return older data. It buys availability and low latency, and costs surprise: a user updates a profile, refreshes, and sees the old one. Naming that user-visible surprise and its usual fix, session stickiness, is what this entry in distributed systems interview questions is really checking.

Answer: Every read returns the latest committed write, as if there were only one copy, no matter which replica serves it. The price is coordination: reads and writes must often wait for a quorum or the leader, so latency rises and, during a partition, a strongly consistent system refuses operations rather than answer stale. Banks pick this; caches usually do not.

Answer: A session guarantee: once you write a value, your own subsequent reads see at least that value, even if other users’ reads may still lag. It is the smallest guarantee that fixes the profile-update surprise while keeping the system otherwise eventually consistent. Common mechanisms are sticky sessions or carrying a timestamp from the write into the next read.

Answer: Any component whose loss takes the whole system down: the one leader with no failover, the one broker, the one DNS provider. The interview expectation is not just naming one, but spotting that eliminating it usually means adding coordination, quorum, or redundancy, each of which reintroduces cost elsewhere. Spotting hidden single points is the skill the question builds toward.

Answer: The classic list of wrong assumptions: the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology does not change, there is one administrator, transport cost is zero, and the network is homogeneous. Interviewers raise it because code written under those assumptions fails in production in exactly the ways, timeouts, retries, and drift, that the rest of distributed systems interview questions explore.

Answer: A deadline a caller sets on waiting for a response. The subtle part, and the reason the question exists: when it fires you know only that the response did not arrive, not whether the other side processed the request. The work may have completed after you gave up, which is why timeouts pair with idempotency rather than blind retries.

Answer: Sending a request again after a failure or timeout. It rescues transient errors, but retries add load exactly when the target is struggling, and if the first attempt actually landed, a naive retry applies the effect twice. That double-threat, amplifying overload and duplicating work, is why every serious answer to distributed systems interview questions on retries includes backoff, jitter, and idempotency keys.

Answer: A small periodic signal each node sends to say it is alive. Silence past a timeout is read as failure. It is the primitive under leader election, membership, and monitoring, and it has a built-in tension: short intervals detect crashes fast but flag healthy nodes during a brief pause, long intervals are safe but slow. Tuning that tension is the follow-up.

Answer: The process of replacing a failed component with a standby: a follower promoted to leader, a standby database taking traffic. What matters in answers is the gap it leaves: detection time, election or promotion time, and, for asynchronous replication, whatever writes the failed primary accepted but never shipped. Automatic failover without addressing that last part is how data disappears quietly.

Answer: Spreads incoming requests across healthy backends, using health checks to route around dead ones and algorithms like round robin or least-connections to share the work. It turns a fleet into one addressable service. The follow-up worth anticipating: a naive balancer can also spread a user’s requests to replicas with different lag, which is where sticky sessions and consistency guarantees re-enter distributed systems interview questions.

Answer: A stateless service keeps no per-request memory, so any instance can serve any call and scaling is just adding instances. A stateful one holds data, sessions, shards, leader roles, so placement matters and failures need recovery procedures. Most architectures push state into dedicated stores precisely so the request-serving tier stays stateless and boring to operate.

Answer: A buffer between producers and consumers that stores messages until consumers are ready. It decouples speed, absorbs bursts, lets a crashed consumer resume where it left off, and turns synchronous dependencies into asynchronous ones. The trade: consumers see messages later, ordering needs care, and delivery guarantees become part of your correctness story instead of the network’s. Queues anchor a surprising share of distributed systems interview questions because almost every architecture has one.

Answer: An operation you can run any number of times with the same effect as running it once: setting an address is idempotent, charging a card is not. It matters because in a distributed system duplicates are unavoidable, timeouts and retries guarantee them eventually. Idempotency keys, a unique token the server deduplicates, are the standard patch, and they appear in distributed systems interview questions on payments, queues, and webhooks alike.

Answer: A partition or key that takes wildly disproportionate traffic, one celebrity user on a sharded platform, one product in a flash sale. Sharding spreads load by key, so a single hot key defeats it: all requests land on one node no matter how many you add. Fixes include caching the value, adding jitter to TTLs, or splitting the key across replicas.

Answer: Vertical scaling buys a bigger machine; horizontal scaling adds more machines. Vertical is simple but caps out and usually needs downtime; horizontal is elastic but forces you to solve partitioning, replication, and coordination, everything else on this page. Interviewers use the comparison to check whether you understand that going horizontal is a design decision, not a deployment option.

Answer: A read that returns an older version of data because the replica serving it has not caught up yet. It is the daily symptom of replication lag: write, look again, see the old value. Tolerable for view counts, intolerable for account balances, which is why answers to distributed systems interview questions here should always ask what the guarantee for that particular read must be.

Answer: A flag stored somewhere shared that says only one worker may touch a resource now, used when several machines could do the same job. The catch that earns credit: the holder can stall or die while still believed alive, so locks need leases with expiry, and truly critical paths need fencing tokens so stale holders are rejected by whatever they write to.

Answer: The mechanism by which a client or proxy finds the current, healthy addresses of a service, since in a dynamic cluster instances start, die, and change IPs constantly. Implementations range from DNS with health checks to registries like Consul or etcd with watchers. Broken discovery masquerades as an outage in one direction only, which makes it a favorite early question.

Answer: A probe a platform runs against each instance, shallow checks that the process answers, or deeper checks that it can reach its dependencies. Load balancers use them to route around unhealthy instances, orchestrators to restart them. The nuance worth adding: a liveness check that depends on a downstream can cascade, one slow dependency makes every instance report dead and turns an incident into a blackout. That cascade risk is why distributed systems interview questions probe what your health check actually tests.

Answer: Designing the system to drop the expensive parts and keep the core when under failure: show cached recommendations but keep checkout working, serve stale data with a banner rather than an error page. It reframes availability as a spectrum rather than a switch. Scenario distributed systems interview questions reward candidates who name which features they would sacrifice first and why.

Answer: A delivery guarantee where a message arrives at least once, possibly duplicated, usually achieved by retrying until the consumer acknowledges. It is the pragmatic default for queues, because losing nothing matters more than doubling rarely. Its direct consequence is that consumers must be idempotent or deduplicate, which is the second sentence interviewers wait for.

Answer: A guarantee where a message arrives zero or one times, never duplicated, achieved by sending without retrying, or tracking exactly what was sent. It is right for telemetry you can afford to lose and wrong for money movement. Stating when each guarantee is the correct choice matters more in distributed systems interview questions than reciting the definitions.

Answer: Because delivery across a network with failures cannot rule out duplicates without an unbounded memory of everything ever sent. What systems actually deliver is effectively-once processing: at-least-once transport plus idempotent or transactional consumers, so duplicates may arrive but produce no duplicate effect. Saying that reframing is the pass; explaining one mechanism that implements it is the distinction.

Answer: The difference between two machines’ clocks at the same instant. Every server keeps its own time, crystals drift, and NTP corrections can step clocks backward. Any logic that orders events by local timestamps alone can mis-order them across machines, which is why distributed systems interview questions use clock skew as the gateway into Lamport and vector clocks.

Answer: A majority of nodes agreeing before the system acts: with three replicas, two must accept a write or answer a read. Majorities are the trick, any two majorities overlap, so two factions cannot both claim victory. That overlap property is what lets quorum systems survive partitions safely, and it is the single most reused fact in this whole bank.

Answer: A small, strongly consistent, consensus-backed store for cluster metadata: who is the leader, which nodes are members, configuration, distributed locks, and watches that push changes. It runs on a few nodes and handles tiny values at low rates. Using it as a database or a message bus is the anti-pattern the question is fishing to expose.

Answer: Keeping hot data in a fast layer, Redis or in-process memory, so most reads never touch the store of record. The catch is invalidation: now there is another copy that can go stale, another failure mode, and a cold-start storm when it empties. Patterns like cache-aside with TTLs and single-flight loading are the concrete answer interviewers want behind the definition.

Answer: The system, from any client that sends more than it can usefully serve, whether malicious, buggy, or just retrying badly. It caps request rates per user or endpoint so overload degrades predictably instead of collapsing everything. In distributed systems interview questions the expected extension is where to count: per-instance limits are cheap but weak, centralized counters are accurate but add a hop and a failure mode.

Answer: Getting several machines to agree on one value, or one ordering of events, even when some crash and messages get lost. It is the formal foundation under leader election, distributed locks, and metadata stores. The honest caveat: consensus needs a majority to make progress, so a partitioned minority side stops deciding rather than deciding alone. That majority requirement is the hinge a lot of distributed systems interview questions swing on.

Answer: Fan-out is one request triggering many downstream calls, a page build fetching a hundred services, or a client retrying to every replica. Availability multiplies down: a page that calls ten services at 99.5 percent each lands near 95 percent. Add retries and the fan-out multiplies load too. This compounding is why distributed systems interview questions keep returning to timeouts, budgets, and bulkheads.

Answer: The guarantee that consumers process messages in the order the producer sent them. They arrive out of order whenever parallelism exists: multiple producers, multiple partitions or shards, or retries re-sending an older message after a newer one landed. The design habit is to scope ordering where it is needed, per key or per partition, instead of demanding global order, which is expensive and fragile. Scoping order rather than demanding it globally is the answer senior distributed systems interview questions listen for.

Answer: Because partitions are imposed by physics and operations, switches die, security groups change, links congest, and no protocol prevents them. The system designer only chooses the reaction: stop serving (keep C) or serve possibly divergent answers (keep A). Interviewers want you to place real systems: synchronous-replica databases choose C and halt or reject, Cassandra-style stores choose A and reconcile later, majority-quorum systems choose C but stay up while a majority survives. Naming a system for each branch is how experienced candidates answer distributed systems interview questions on CAP.

Answer: PACELC says that even without partitions (Else) you trade latency against consistency: replicate synchronously and every write waits, replicate asynchronously and reads can lag. It matters because the Else branch is your everyday behavior; CAP only describes the rare, bad day. The practical sentence that lands: every consistency knob in a real store, quorum sizes, sync replicas, wait-for-ack settings, is a PACELC dial, and distributed systems interview questions on tuning are really asking which side of that dial you chose and why.

Answer: N is replicas per key, W the acks a write needs, R the acks a read needs; W + R > N guarantees read and write sets overlap, so a quorum read sees the latest quorum write. W = N, R = 1 gives fast reads and fragile writes; W = 1, R = N the reverse; W = R = 3 over N = 5 is the balanced default. The caveat that earns credit: quorum overlap preserves recency only if the coordinator compares versions and returns the newest, and slow replicas still make the p99 the quorum, not the average.

Answer: The leader serializes writes into a log, a WAL or binlog, and followers apply it in order. Synchronous replication waits for one or more followers to acknowledge before confirming to the client, so a leader crash loses nothing but every write pays the round trip. Asynchronous confirms immediately and ships in the background, so failover can lose the unshipped tail. The production middle ground, semi-sync, waits for one follower, bounding data loss to one acknowledgment window.

Answer: The delay between a write on the leader and its application on a follower, caused by bursty write volume, a single-threaded applier, or a follower busy serving reads. Measure it at the source of truth: Postgres exposes replay lag in pg_stat_replication, MySQL reports seconds behind in SHOW REPLICA STATUS, Kafka tracks consumer lag per partition. Route reads away from replicas whose lag exceeds a threshold, and alert on the lag distribution, not the average, because a five-minute tail is the incident. Lag arithmetic like this is bread and butter for intermediate distributed systems interview questions.

Answer: Several nodes accept writes, typically one per region or datacenter, forwarding changes to their peers, which cuts write latency across distance and survives one leader dying. The new problem is write conflicts: two leaders accept incompatible updates to the same record concurrently and both commit locally. Everything else, conflict detection, resolution strategy, and user-visible merge surprises, flows from that, which is why distributed systems interview questions about multi-region setups almost always land here.

Answer: Three families. Conflict-free datatypes, CRDTs like counters and LWW-register sets, converge mathematically no matter the order. Application-level merge keeps both versions as siblings and lets the application or user reconcile, which Dynamo exposed and CouchDB made explicit. Automatic policies like last-write-wins silently discard one side. Strong answers mention keeping the versions and resolving deliberately, and that resolution logic must be deterministic on every replica or the replicas diverge anyway.

Answer: LWW picks whichever write carries the larger timestamp, which silently discards the loser, and under clock skew the loser can be the newer edit, a node with a fast clock overwrites slower nodes forever. It is tolerable only where writes commute or staleness is cheap, like profile fields. For counters, cart merges, or concurrent document edits it destroys data without any error, and surfacing that silent-destruction property is the point of the question.

Answer: Place nodes and keys on a hash ring; each key belongs to the first node clockwise from its hash. Adding or removing a node moves only the keys in that arc, roughly one over n of the dataset, instead of rehashing everything as modulo-sharding does. Replication takes the next nodes clockwise for the preference list. Ring fluency shows up in distributed systems interview questions about sharding and cache clusters alike; expect the follow-up on hot spots, since a plain ring cannot fix one celebrity key.

Answer: Each physical node appears on the ring many times, as dozens or hundreds of virtual nodes. That evens out the luck of the draw, with few physical nodes a raw ring gives wildly uneven arcs, and makes rebalancing smooth: a new node picks up many small slices instead of one huge neighbor’s arc. It also lets heterogeneous hardware take proportionally more of the ring, which is how Cassandra and Dynamo clusters express weight.

Answer: Range partitioning assigns contiguous key ranges to nodes: range scans and time-series drops are efficient, but sequential keys concentrate on one partition, the classic hot-shard on insert. Hash partitioning spreads keys by hash, giving even write load but scattering ranges, so a scan of one range touches every node. The judgment the question wants: pick by access pattern, ordered queries and retention jobs favor ranges, uniform key-value load favors hashes.

Answer: Rebalancing moves data between nodes while serving traffic: it consumes I/O and network, doubles-cache misses in flight, and mistakes during it cause missing or duplicated ranges. Fixed-size partitions soften it: split the keyspace into many more partitions than nodes, then moving a partition is metadata plus a copy, not a rehash of every key. Pair that with rate-limited data movement and double writes during handover, and rebalancing becomes boring instead of terrifying.

Answer: Skew is load or data volume concentrated on a few partitions, the tenant with a huge dataset or the key everyone reads. Treatments: split the hot range and spread the halves, salt composite keys by appending a suffix so one logical key becomes several physical ones, cache the hot value, or give the tenant a dedicated shard. The monitoring sentence to include: per-partition, not per-cluster, metrics are how you see skew before it pages you.

Answer: Followers that hear nothing from the leader past a randomized election timeout become candidates: they increment the term, vote for themselves, and request votes. A candidate wins with a majority and immediately sends heartbeats to assert leadership; a peer refuses to vote if it has already voted this term or if the candidate’s log is less up-to-date than its own. Randomized timeouts stop split votes from repeating, and at most one leader exists per term, which is the invariant everything else rests on.

Answer: A monotonically increasing logical epoch, one per possible leadership. Every message carries a term; a node seeing a higher term immediately updates itself and steps down if it was leader, and a leader seeing a higher term in any response knows it is stale. Terms are a logical clock for authority: they let the cluster detect stale leaders without trusting wall clocks, which is the sentence distributed systems interview questions on split-brain prevention are fishing for.

Answer: The leader appends a client command to its log and sends AppendEntries to followers, which append and acknowledge. Once the entry sits on a majority, the leader commits it, applies it to its state machine, and confirms to the client; followers learn commit on the next exchange. Entries only from old terms commit indirectly, when a current-term entry commits alongside them, which is why Raft leaders append a no-op at term start. Uncommitted entries from a deposed leader can be overwritten; committed ones cannot, because any electable candidate carries every committed entry.

Answer: Paxos agrees on one value with safety despite crash failures and lost messages: at most one value is chosen, and a chosen value never changes, using two phases of numbered proposals where a majority acceptance makes it final. Raw Paxos is single-value and slow, so Multi-Paxos elects a stable proposer that skips the first phase for subsequent entries, turning it into a replicated log. The honest interview line: few teams run textbook Paxos, but Raft, ZAB in ZooKeeper, and every consensus store are its engineering descendants.

Answer: A lock service hands out a monotonically increasing token with each grant, and the storage rejects any write whose token is older than the newest it has seen. That closes the zombie-lock hole: a paused or partitioned holder wakes, still believing it holds the lock, and writes anyway. Tokens turn stale holders into rejected writes instead of corrupted data. ZooKeeper’s zxid and etcd’s revision serve this role, and naming the mechanism is what this question in distributed systems interview questions is designed to surface.

Answer: A per-node counter advanced on every local event, sent along with each message, and bumped to max of local and received plus one on arrival. They guarantee the arrow of causality: if a happened before b, then the clock of a is less than the clock of b. The converse fails, equal or ordered timestamps do not prove causation, and concurrency is undetectable, so Lamport clocks order but cannot find conflicts. Adding a node id tiebreaker gives a total order, which is exactly how leaderless stores build deterministic sort keys.

Answer: A vector with one counter per node, merged element-wise on message receipt and incremented locally per event. Comparing two vectors yields three outcomes: one dominates, which gives a causal order, or neither dominates, which proves the two events happened concurrently. That concurrency proof is the superpower: Dynamo-style stores keep sibling values when vectors conflict instead of guessing. The cost is size proportional to nodes and the human cost of resolving the conflicts it finds, which is why stores make LWW an opt-out.

Answer: A timestamp with a physical part and a logical part: the physical half tracks the wall clock and never goes backward, and the logical half increments when events arrive out of order or the clock stalls. You get timestamps that look like time, useful for TTLs, range scans, and humans, plus Lamport-style causality and monotonicity. CockroachDB is the flagship user, stamping transactions with HLCs so that reads are consistent without waiting out clock uncertainty the way Spanner does.

Answer: TrueTime exposes time as a bounded interval, earliest to latest, backed by GPS and atomic clocks in every datacenter, instead of a single lying instant. Spanner uses it to order transactions against real time: a write commits only after waiting out its timestamp’s uncertainty window, which makes the observed order match wall-clock order, external consistency, linearizability with real time. The trade is explicit and instructive: correctness purchased with a per-commit wait, roughly the uncertainty width, typically single-digit milliseconds.

Answer: A guarantee that everyone sees causally related operations in the same order, a reply must never appear before the message it answers, while concurrent operations may be seen in different orders on different replicas. It sits between eventual and sequential: stronger than eventual because related events cannot invert, weaker than sequential because there is no single global order. Comment threads and collaborative state fit it well, and implementing it needs causality tracking, vector clocks or session contexts, which is why distributed systems interview questions often pair causal consistency with the clock topics above.

Answer: Four: read-your-writes, monotonic reads so you never see time go backward, monotonic writes so your writes apply in the order issued, and writes-follow-reads so a reply cannot be overwritten before you see it. Implementations attach a small token to the session, a timestamp or version vector, and route or reject reads that would violate it. Sticky routing to one replica is the cheap version, token comparison is the portable one, and naming all four is what separates book answers from practiced ones here.

Answer: Two paths, chosen by the gap. If the leader’s log still covers the follower’s position, it reconnects and replays the missing WAL segment, which is the cheap case. If the gap exceeds retention, the log was truncated, so restore a base backup or snapshot, then replay forward from a recorded position, in Postgres a backup plus restore_command, in Kafka a full partition re-fetch. Operational rule: size WAL retention for your worst realistic outage, because catch-up is only as cheap as the log you kept.

Answer: Fixed timeouts are brittle: one threshold misjudges every network. Phi-accrual detectors, used by Cassandra and Akka, model each heartbeat’s inter-arrival distribution and output a suspicion score, so flaky links get more tolerance than clean LAN links and the same tuned threshold adapts across conditions. The detection decision becomes how sure, not how long. Name the inputs honestly: heartbeat interval, network jitter, GC pauses that look like crashes, because misdetection triggers elections, and elections are expensive. That cost framing is what intermediate distributed systems interview questions on detection are really about.

Answer: Nodes periodically pick random peers and exchange state, membership, load, schema versions, merging what they know. Information spreads exponentially, the whole cluster converges in logarithmic rounds, no coordinator exists to be a single point of failure, and partial connectivity still converges eventually. The costs are eventual, not immediate, agreement and background network chatter. Cassandra’s membership and Consul’s SWIM implementation are the standard references to cite in distributed systems interview questions on this topic.

Answer: Anti-entropy is the background machinery that converges replicas that missed writes. Read repair compares the replicas consulted during a quorum read and pushes the newest value to laggards, fixing divergence exactly where it was discovered. Hinted handoff covers a replica that was down: the coordinator keeps the write locally with a hint and delivers it when the node returns. Both are best effort, which is why the deeper repair below exists, and saying that sentence is what the question is really listening for.

Answer: Comparing two replicas byte by byte is prohibitive, so each side builds a Merkle tree, a hash tree where leaves hash key ranges and parents hash their children. Comparing roots tells you whether anything differs; descending only into mismatched subtrees locates the diverging ranges in logarithmic comparisons instead of full scans. Cassandra’s nodetool repair builds these per range. The trade to mention: trees are expensive to build, so repairs are scheduled, throttled operations, not a response to every hiccup.

Answer: A coordinator drives two rounds. Voting: it asks every participant to prepare, each writes its changes and a prepare record to durable storage, locks the rows, and votes yes or no. Commit: on unanimous yes the coordinator records the decision and tells everyone to commit; any no, or any timeout, aborts everywhere. It is atomic because the decision is written down before it is applied, and it is the only mainstream way to get atomic commit across heterogeneous stores without a consensus system underneath. The blocking risk is also the bridge to the next wave of distributed systems interview questions about sagas.

Answer: The bad moment is the coordinator crashing after participants voted yes but before the decision lands: every participant holds locks and sits in-doubt, unable to commit or abort, blocking everything it touched until the coordinator recovers and announces the outcome. A participant crashing after prepare blocks the round too. That blocking is why 2PC is called a blocking protocol, why coordinator logs matter, and why modern databases replace it with consensus-backed replication or push atomicity up into sagas, which is the comparison distributed systems interview questions are circling.

Answer: A saga chains local transactions, each committing independently, and pairs every step with a compensating transaction that undoes it semantically, reserve stock with release stock, charge with refund. Backward recovery runs compensations in reverse when a step fails, restoring business-level consistency without distributed locks. Forward recovery retries the failed step until it succeeds, for steps that cannot fail permanently in practice. It trades isolation for availability: intermediate states are visible, so designs add idempotency and, where needed, countermeasures like semantic locks.

Answer: It solves the dual-write hole: a service commits a database row and publishes an event, and a crash between the two produces a database without its event, or an event for a rolled-back change. The outbox writes the business row and the event row in the same local transaction, then a relay tails the outbox table and publishes, marking rows sent. Atomicity comes from the one real transaction you had, no 2PC, and idempotent consumers absorb the relay’s at-least-once delivery.

Answer: The mechanism by which a slow consumer tells upstream to slow down, instead of buffering without bound until memory dies. Forms: bounded queues whose puts block or reject, pull-based consumption where workers fetch at their own pace, TCP flow-control windows, and explicit signals like reactive streams or gRPC flow control. The design rule underneath: buffer size is a choice, choose a small one and propagate delay, or choose infinity and choose collapse, and distributed systems interview questions on stability are really asking which you chose.

Answer: A message whose processing always fails, malformed payload, or an invariant the consumer cannot satisfy, so redelivery turns it into an infinite crash loop that blocks the partition. The countermeasure is a retry policy with a cap, after which the broker parks the message on a dead-letter queue and moves on. The DLQ is not a trash can, it is an investigation queue with alerting, triage, and a replay path once fixed. Without the cap, one bad message takes down the pipeline, which is the story this question wants. Every pipeline meets a poison message eventually, which is why distributed systems interview questions treat DLQ design as a proxy for experience.

Answer: When concurrent writes race, the coordinator cannot pick a winner without inventing an order, so it stores both versions together with their vector-clock context. A read returns the sibling set plus context, and resolution happens either in the client, which merges deliberately, or via a fallback policy like LWW that discards data silently. The context matters: submitting a resolved value with the old context proves causality, letting the store collapse the siblings. That read-modify-write with context is the mechanism interviewers want named.

Answer: Black-box consistency checking, the Jepsen method: drive concurrent operations while injecting partitions, clock skew, and crashes, record every call with real times and results, then check the history against the claimed model, linearizable, causal, serializable. In-process alternatives are deterministic simulation, which replays random schedules bit for bit until a bug is reproduced and fixed. Both exist because unit tests cannot see interleavings, and stating that interleavings are the bug source is the insight this question rewards.

Answer: Because requests amplify tails: a page backed by a hundred services at 1 percent slow each is likely slow overall, since every user request rolls the dice on every backend. Variance sources compound, GC pauses, queueing, thermal throttling, a laggy replica. Mitigations: fan-out reduction, hedged requests that send a duplicate after a short delay, tail-aware load balancing that avoids the slowest replica, and per-request deadlines with budgets. The sentence to land: at fleet scale, p99 is what users experience, so distributed systems interview questions about p50 numbers are usually traps.

Answer: A topic splits into partitions, each an append-only log with a single writer order, and consumers process each partition strictly in offset order. Ordering is therefore per key, since keys hash to partitions, and per partition only. It breaks when partition count changes mid-flow, a key moves and its order splits, when multiple producers retry without idempotence and interleave duplicates, or when a consumer fans work out to threads that ignore offsets. Scope the order you promise, then refuse to resize casually.

Answer: By stacking mechanisms: an idempotent producer tags each batch with a producer id and sequence so broker retries cannot duplicate; transactions make producing to several partitions atomic and let consumers read only committed data; downstream, deduplication tables or idempotency keys collapse anything that still slips through. Kafka’s exactly-once semantics is this stack, not magic delivery. The general law to state: transport stays at-least-once, effects are made once through idempotence, and interviewers probe whether you know the difference. That law is the exact dividing line intermediate distributed systems interview questions draw on this topic.

Answer: First verify it is true split-brain and not perception: check terms or epochs on both nodes, etcdctl endpoint status --cluster -w table or Raft status endpoints, and the quorum size. A healthy consensus system cannot keep two leaders, so a stale leader with an old term still serving writes means something bypassed the protocol: a node outside quorum with a client pinned to it, or an application-level lock without fencing. Remedy: kill traffic to the lower-term node, force it to catch up or rejoin, and audit every write path that does not carry a term or fencing token. The diagnostic order, terms first, then quorum membership, then bypass paths, is the entire answer.

Answer: Clock skew before application bugs. Compare chronyc tracking offsets across the nodes that wrote those records, and look for a step correction or an unsynced VM whose host throttled. If ordering logic consumes wall timestamps from different machines, skew of a few hundred milliseconds inverts the story. The fix is layered: repair NTP, then stop trusting local timestamps for cross-machine order, stamp events with a logical or hybrid clock at a single serialization point, or keep a causal token with the event. State the diagnostic order, offset check, then step history, then schema, because distributed systems interview questions like this one are tests of where you look first.

Answer: Diagnose first: confirm only W-minority replicas remain, and whether reads still work with R reachable. Then the decision the round is really about: a quorum system refusing writes is correct, it is protecting consistency, and forcing it down a notch, dropping W below majority, trades away exactly the guarantee you run it for. Options in order: restore connectivity or capacity, fail over to a region that holds majority, and only for a declared, time-boxed emergency, lower the consistency level while alerting every consumer that stale and lost writes are now possible. Saying what you refuse to do by default is what this question in distributed systems interview questions scores.

Answer: Thundering herd, twice. During the failover window every client timed out, and all of them retried on the same schedule, so the recovered primary meets its full backlog at once while every cold cache key misses to the database simultaneously. Countermeasures in order: exponential backoff with jitter so retries spread out, circuit breakers that fail fast while the primary warms, single-flight or request coalescing so one cache miss per key loads instead of a thousand, and gradual traffic ramp after failover. Without those, the recovery becomes a second outage, which is why distributed systems interview questions keep this pattern in heavy rotation.

Answer: Nothing silent, that is the first answer. A failed compensation leaves the saga in an explicitly incomplete state, so the correct behavior is to persist the failure, keep the compensation in a retriable queue with backoff, alert a human with the context to decide, and expose the order’s state to support and the customer. Then dig at the design: compensations must be idempotent and retriable, so check whether this one was a one-shot call with no durability, and whether the provider outage should gate the saga from proceeding that far, a prior-step health check or a reservation-based flow. The graded sentence: compensation failure is an operations state, not an exception to swallow. Interviewers rank saga failure handling among the most revealing scenario distributed systems interview questions precisely because it has no happy path.

Answer: Replication lag, almost certainly amplified: the import saturated the leader, followers fell minutes behind, and reads routed to them returned pre-import state, so data looked deleted. Verify in order: lag metrics on every replica, pg_stat_replication or the store’s equivalent, the read routing configuration during the import window, and whether any user hit a lagging replica immediately after writing, a read-your-writes violation. Fixes: throttle imports, route the importing user’s session to the leader or a fresh replica, enforce session guarantees with a token, and alert on lag per replica rather than in aggregate. The appearance-then-vanish story is what makes this a classic scenario distributed systems interview question.

Answer: Leader flapping means election triggers recur, so hunt the causes in order. First heartbeat path: packet loss or a saturated NIC making followers miss heartbeats, check retransmits and drops. Second, pauses on the leader itself: GC stop-the-world, swap, or CPU starvation, correlate election timestamps with GC and load graphs. Third, configuration: election timeouts shorter than your worst pause, one slow disk flush and the node times out. Fourth, membership churn, a flapping member resetting the quorum repeatedly. Fix the underlying pause or network, then widen randomized timeouts to exceed worst-case pauses, and check the log for the term history to confirm the flap frequency matches the trigger. This is one of those distributed systems interview questions where the log correlation, not the config edit, is the answer.

Answer: Triage space first, because the leader dying makes everything worse: remove WAL the standby can never consume, after confirming the follower cannot catch up in place, or add space, whichever is faster. Then decide the recovery path: if the gap exceeds WAL retention, drop the replica and rebuild from a fresh base backup with a current position, throttled so the leader survives it, and keep the replica out of the read pool until it converges. Prevention is the closing point: alert on lag and on WAL retention growth separately, because they are different failures, and size retention for the worst realistic outage you intend to ride out without a rebuild.

Answer: Isolate before touching anything: is it that shard’s leader, its replicas, or the clients routed there? Check the shard’s host for saturated disk or CPU, then the shard’s data shape, since one hot key or one oversized partition turns latency into timeouts while the rest of the fleet idles, compare per-shard QPS and item counts. Then check placement: the busy shard collocated with a batch job or a backup stream. Remedies follow the cause, split the hot range, cache or salt the hot key, move noisy neighbors, or rebalance partitions. The diagnostic principle, per-shard metrics first, is what this entry in distributed systems interview questions exists to test.

Answer: New nodes are innocent until proven guilty, so check the rebalance first: streaming copies consume I/O and network, evict cache, and can trigger full compactions, which alone explains the slowdown, and it should end when rebalancing completes. Then check ring movement: if the partitioning scheme rehashed keys instead of moving fixed partitions, most of the dataset moved at once. Third, check consistency settings: adding replicas can raise effective quorum sizes or change the preference list so reads now cross racks. Finally, confirm the new nodes are actually warm and healthy before judging, and rate-limit data movement next time. The lesson, addition is a load event, is the point of the question.

Answer: Diagnose which half is broken: consumers stuck, throughput per consumer collapsed, or producer rate genuinely spiked. If consumers are stuck, look for a poison message looping without a DLQ cap and pull it. If throughput fell, scale consumers up to the partition count, since more consumers than partitions idle, and profile the handler for a slow dependency. If load is real, producers need backpressure: apply rate limits, degrade by dropping low-priority messages at publish time, and only then grow the cluster. Bound the queue regardless, because unbounded depth converts overload into OOM deaths, and say that sentence out loud, it is the one this scenario distributed systems interview question is built around.

Answer: The retry fired after the first request actually succeeded but its response was lost, so the timeout lie became a duplicate charge: retries without idempotency are duplicates waiting to happen. Immediate fix is procedural, reconcile against the provider and refund. Real fix is structural: generate an idempotency key per business attempt, send it on every retry, and make the provider or a local dedup store collapse repeats, plus make the charge operation check-then-act on a recorded intent so it is safe to re-run. Prove it with a test that kills the response after commit. Retrofitting idempotency under incident pressure is the scenario interviewers love, because it touches correctness, money, and diagnosis at once, and it remains one of the most common failure stories distributed systems interview questions draw from.

Answer: First identify the violated assumption: someone promised global order while the transport only gives per-partition order. Options in cost order: co-locate the related events on one partition by keying them together, which restores order where it is needed at the cost of parallelism; add a version or sequence number to events and make the consumer ignore or buffer stale versions, which fixes state without moving data; or move the ordering-critical path to a single ordered stream and keep the rest parallel. Do not resize partitions while ordering matters. Naming per-key ordering as the standard compromise, and its resize caveat, is what completes this answer.

Answer: etcd without a majority of 2 of 3 refuses every write and, depending on config, reads, so leases expire, leader election for controllers stalls, and the control plane appears dead. Recovery: restore the third member or network path so a majority exists again, in that order of preference, because a two-member cluster is one failure from data loss. If a member is truly gone, remove it so the cluster reconfigures to a healthy quorum of the survivors. Operational lessons to state: size etcd for odd membership, never do rolling maintenance that dips below majority, snapshot it on schedule, and alert on quorum health, not node count.

Answer: Treat it as a unresolved conflict, not sync delay, once lag is ruled out: check replication lag between leaders first, then look for the two conflicting writes in the store’s version metadata. If they exist as siblings or one was discarded by last-write-wins, that is the answer, a clock-skewed region won LWW and overwrote newer intent. Immediate remediation rewrites the correct value with current context; durable fix is to stop discarding, keep versions, resolve deterministically, and if business semantics cannot automate it, route the field through a single leader or expose resolution to the user. This is the scenario form of the multi-leader conflict question, and the version metadata inspection is what proves competence.

Answer: Anchor the period first: graph latency against candidate timers, leader election logs, compaction or checkpoint schedules, snapshot jobs, TTL sweeps, and cron-style maintenance. Then correlate with resource signals, disk write bursts, network saturation, GC pause logs on the JVM services. The usual suspects for a minutes-scale period are a heartbeat or election cycle disturbed by a recurring pause, log compaction, and a scheduled job whose lock makes every node wait. Fix by decoupling: spread the job across time with jitter, isolate its I/O, and tune pause sources. The transferable skill, matching periods to schedules before touching configs, is exactly what distributed systems interview questions at this level reward.

Answer: A monotonic reads violation: successive reads in one session move backward in time because a load balancer sent the second request to a more-lagging replica. Eliminate it three ways, cheapest first: sticky sessions so one user lands on one replica, which has its own balancing downsides; a session token, the timestamp or version from the first read, carried on later reads so any replica can check it is current enough; or route all of a user’s reads to the leader for a grace period after any write. Distributed systems interview questions phrase it as the avatar story precisely to see whether you reach for the token mechanism rather than shrugging at eventual consistency.

Answer: The participants are in-doubt: they voted yes, durably recorded prepare, and now await a decision only the coordinator’s log can give, so the runbook is recover the coordinator, not the participants. Bring the coordinator back from its persistent transaction log, which replays the pending decision and lets it announce commit or abort; until then anything holding those locks will block, so expedite the coordinator’s restart above all else. If the coordinator’s disk is gone, you are in manual resolution: inspect each participant’s prepare record and decide per transaction, then redesign afterward, sagas or a consensus-backed coordinator, so no single node’s crash can freeze the fleet. Coordinator recovery discipline is a classic of scenario distributed systems interview questions for exactly this reason.

Answer: Work the chain in order: is the node receiving writes at all, hinted handoff queue full, dropped, or expired, is streaming throttled to a trickle, and has any repair actually run, since read repair only fixes keys that happen to be read. Old values being served means the node is in the read path before convergence, so first pull it from serving until it catches up, then force repair over the affected ranges and watch streaming metrics, not hope. Prevention: cap hint lifetime honestly, alert on divergence estimates, and gate rejoin behind catch-up completion. The error to name, admitting an uncaught-up replica into the read path, is the heart of this scenario distributed systems interview question.

Answer: A tail-only event, so hunt what hits a minority of requests at a scheduled time: a batch job or backup saturating one disk or shard, a cache TTL expiry wave refilling en masse, a compaction or GC cycle on some nodes, or a scheduled report scanning cold data through the serving path. Confirm by decomposing: per-node, per-shard latency for that hour, which turns a global p99 into a named offender, then check whether slow requests correlate with specific keys or specific replicas. Mitigate by isolating the job’s I/O, jittering TTLs, and hedging or bounding the affected reads. The discipline, slicing p99 by node and key before tuning anything, is the move senior distributed systems interview questions wait to hear.

Answer: The gap is trusting a lease as truth on the holder’s side: the worker paused or lagged, its session expired, ownership moved, and the stale holder kept going because nothing rejected it. Diagnosis confirms two writers with overlapping intervals in audit logs. The fix is fencing: every lock grant carries a monotonic token, and the resource being protected, the storage layer, refuses writes whose token is older than the newest seen. Without a fenceable resource, add the check inside the transaction that claims progress, compare-and-swap on a version. This question, and its follow-ups, is a staple of scenario distributed systems interview questions because it separates people who have run locks from people who have read about them.

Answer: Raise read consistency at the read path first: use quorum reads with a compare-and-return-newest coordinator so readers stop seeing divergence, accepting the latency cost, and enable read repair so every such read converges what it touches. Check why repair fell behind, backlog, throttling, or a member down, and run targeted repair on the hot ranges rather than the whole ring. Then address the source: a node dropping writes or a partitioned replica generating the divergence, which per-replica drop metrics will show. The ordering, protect readers, converge opportunistically, then fix the writer, is the sequence this scenario distributed systems interview question rewards.

Answer: A rebalance storm: consumers fail heartbeats or poll too slowly, the group coordinator evicts them, partitions reassign, processing stops, the lagging consumer catches up, and the cycle repeats. Diagnose by reading the coordinator logs and consumer metrics, look at poll interval versus max.poll.interval.ms and session timeout, and at stop-the-world pauses or long synchronous calls inside the poll loop. Tune honestly: heartbeats and session timeout sized to your network, max poll interval above the worst handler latency, or move heavy work off the poll thread. Also check for deployments, restarting all consumers at once triggers the same storm, so roll them gradually with jitter. Deployment-induced rebalances are the version of this story most distributed systems interview questions quietly assume.

Answer: Say the truth first: asynchronous replication means an acknowledged write can be lost if the leader dies before shipping, this was a known RPO, and the incident revealed it. Then the meeting-worthy changes, in cost order: make replication semi-synchronous so commit waits for one follower, acknowledge only after the log is durable on the leader with sync flush, shorten the ship window, and quantify what the business actually tolerates, because full synchronous everywhere may cost more latency than the data is worth. Add a reconciliation process for the gap window regardless. Being able to state RPO versus RTO tradeoffs calmly is exactly what this scenario distributed systems interview question tests.

Answer: Nothing crashed incorrectly, the design made liveness wait on worst-case death detection: a long lease protects against false expiration but turns every real crash into a long stall. Redesign on two axes. Shorten the stall: renew leases by heartbeat so the TTL only needs to cover one missed renewal, cutting detection from minutes to seconds. Remove the single-owner stall entirely where possible: make the work idempotent and let a contender take over with a fencing token, or replace the lock with optimistic concurrency, claim a version, do the work, commit if unclaimed. The trade to articulate, detection speed versus false expiration, is the entire lesson of this question.

Answer: It tells you the drill found a real consistency bug: a correct quorum system lets only the majority side accept writes, so dual acceptance means W was below majority, clients bypassed the coordinator, or the store’s protection was disabled. Verify in order: effective write consistency settings at the moment of the test, whether each side actually has a majority, membership and quorum math, then the client path for direct-to-replica connections or stale routing tables. Repeat the drill while watching terms, ack counts, and read results from a witness client on each side. Publishing the drill’s findings and the fix, is exactly the behavior senior distributed systems interview questions about operational maturity are probing.