Mooncake Explained: KV-Cache-Centric LLM Serving

Serving engines spent years optimizing the transformer forward pass, and then discovered the real bottleneck was moving context around. Mooncake, open-sourced by Moonshot AI’s infrastructure team, is the clearest expression of that discovery: a serving architecture designed around the KV cache itself, where prefill and decode live on separate machines and cached context is a pooled, transferred resource rather than an accident of GPU memory. This explainer covers the idea, the numbers, and why it matters for where inference is going.

The work earned the Best Paper Award at FAST 2025, storage and systems research’s premier venue, which is itself a signal: the interesting inference problems have become data-movement problems. Our serving engines overview covers the engines; this piece covers the layer underneath them that is quietly reshaping how they are deployed.

The problem: prefill and decode want different machines

An inference request has two phases with opposite appetites. Prefill, digesting your prompt into KV cache, is compute-bound and benefits from big batches and full GPUs. Decode, emitting tokens one at a time, is memory-bound and prefers small batches and low latency. Colocating both on one cluster forces each phase to tolerate the other’s constraints, so long prompts stall token generation and chatty sessions waste compute capacity. Operators see it as the twin spikes problem, throughput requests and latency requests punishing each other on the same hardware.

Disaggregation is the fix with precedent: databases separated compute from storage years ago for the same reason. Mooncake applies it to inference by splitting prefill and decode onto separate clusters, then answering the question that split creates, how does the context produced on one machine reach the other, with its centerpiece: a cache built to be moved.

The idea: KV cache as a pooled, transferable resource

Mooncake’s second move is scavenging capacity that serving clusters already own. GPU machines carry substantial CPU, DRAM, and SSD that sit largely idle while the GPUs work; Mooncake aggregates that stranded capacity into a distributed KV cache pool, tiered across memory technologies, separate from any single request’s lifetime. Cache becomes infrastructure with its own lifecycle, and GPU memory becomes a fast tier in a hierarchy rather than the only shelf.

The consequences ripple. Prefill results stream to the pool instead of pinning a GPU. Decode nodes pull exactly the context they need. Repeated prefixes, multi-turn conversations, shared system prompts, become cache hits served from the pool rather than recomputed on silicon that costs orders of magnitude more per byte touched. The architecture is honest about its own name: KVCache-centric is a literal design statement, not branding.

The Transfer Engine: 190 gigabytes per second of boring reliability

None of that pooling matters if moving cache costs more than recomputing it, which is why the Transfer Engine is the component that won the paper award. It moves data across the heterogeneous networks inside a cluster, aggregating bandwidth across multiple RDMA interfaces, choosing paths with topology awareness, NUMA affinity included, and routing around transient network failures automatically. On a forty-gigabyte transfer, roughly the KV cache a 128-thousand-token prompt generates for a 70-billion-parameter model, it sustains up to 87 GB/s on four 200-gigabit links and 190 GB/s on eight 400-gigabit ones, several times what TCP delivers on the same fabric.

Those numbers are the difference between disaggregation as a slide and disaggregation as a system. A pool whose fills arrive faster than recomputation would finish is a pool you can actually depend on, and the failover behavior is what makes it deployable in production rather than a benchmark.

What the production numbers say

The headline figure comes from real traffic: under Kimi’s workloads, Mooncake’s architecture handles 75 percent more requests while staying inside latency objectives. The mechanism is the compounding of both moves, prefill no longer interfering with decode, and cache reuse eliminating recomputation for repeated context, and the effect is multiplicative rather than additive.

Two pieces of supporting evidence raise confidence. The benchmark traces behind the FAST paper are publicly released, so the claims are reproducible rather than atmospheric. And the integrations are official rather than forked: vLLM adopted the Transfer Engine for disaggregated prefill and cache transfer first, then SGLang followed, meaning the major open engines now treat this layer as infrastructure they plug into.

What is in the repository

The project ships as separable components rather than a monolith. The Transfer Engine stands alone for teams that only want fast cross-node data movement. Mooncake Store extends it into distributed KV cache and model-weight management. The elastic-MoE serving components address expert-parallel serving for mixture-of-experts models, where the disaggregation logic gets another workout routing expert activations. Documentation covers SGLang and vLLM integration as first-class deployment paths, and there is even a skills package so AI coding assistants can configure Mooncake deployments from their own tooling, a small but telling detail about who the expected operators now are. The whole repository is Apache 2.0 C++ from the kvcache-ai organization, with the releases of Store, Transfer Engine, and the elastic serving components staggered across 2025 in a sequence the updates log records faithfully, useful reading for anyone tracing which pieces are oldest and therefore most battle-tested.

Common misconceptions

Misconception one: Mooncake is a serving engine competing with vLLM. It is the layer engines integrate; both major engines run on top of it. Two: disaggregation is only for hyperscale. The architecture pays wherever long prompts and latency-sensitive decoding coexist, which includes modest clusters serving agentic workloads. Three: the cache pool eliminates the need for prefix caching in the engine. They compose, with the pool extending cache lifetime and reach beyond a single node’s memory, not replacing per-node caching. And a fourth misconception worth naming: that 75 percent figure is a universal speedup. It is a capacity gain under Kimi’s traffic mix, on their prefix-repetition profile and their fabric, and your workload’s shape decides your number.

Pro Tips

• Read the FAST 2025 paper before the code; the architecture diagram is the map for every component.
• Start with the Transfer Engine alone if your need is cross-node data movement, not full disaggregation.
• Check your network before anything else: RDMA capability determines whether pooling pays at your scale.
• Measure your workload’s prefix-repetition rate first; it bounds the win cache pooling can deliver.
• Follow the released traces to see what production conversation traffic actually looks like before modeling your own.

Further reading

• The Mooncake repository: components, integration guides, and the released traces.
• Our inference optimization primer: where KV-cache economics sit among the other levers.
• Our inference learning path: the systems background this architecture assumes.

Why did a storage conference give this the best paper award?

Because Mooncake reframed an inference problem as a storage and data-movement problem, and then solved it with storage engineering. The KV cache is, mechanically, a large stateful dataset with an access pattern: written once during prefill, read repeatedly during decode and across sessions, evictable by recency and reuse. That is exactly the shape storage systems have optimized for decades, tiering across memory technologies, pooling across nodes, and moving data asynchronously behind the request path. FAST recognizing the work is the field acknowledging that LLM serving’s hardest scaling problems are not model problems but data-management problems, and the award-winning substance is the unglamorous part, topology-aware transfer, failure-tolerant routing, tier scheduling, rather than any model-side novelty. For practitioners the lesson generalizes: when a new workload emerges, ask what its data looks like as a storage problem, and the answer frequently reshapes the architecture.

When does disaggregation pay off for a smaller deployment?

The threshold is workload-shaped more than scale-shaped. The signature that pays is long prompts paired with latency-sensitive generation: agentic loops re-reading large contexts, document assistants, coding agents with repository-scale context. If your traffic is short prompts and short answers, colocated phases cost you little and disaggregation buys mostly complexity. If your prompts run to tens of thousands of tokens while users watch token latency, the interference between phases is your dominant inefficiency, and splitting clusters pays even at modest node counts. The second factor is prefix repetition: multi-turn conversations and shared system prompts make the cache pool earn its keep, while unique-context workloads get less from pooling and more from the phase split alone. Size matters mainly through the network, since RDMA fabric quality decides transfer economics, and small clusters with modern networking often have better fabric per node than the big ones. A useful planning exercise: estimate your fleet’s stranded DRAM and SSD capacity, multiply by your prefix-repetition rate, and compare the result against your GPU compute bill. The ratio tells you whether to pursue pooling this quarter or next year.

How does Mooncake relate to what vLLM and SGLang do natively?

The engines own the serving loop, batching, scheduling, paged KV management inside a node, and their native innovations, continuous batching and paged attention, remain the foundation everything sits on. Mooncake takes over where the single node ends: cross-node cache placement, transfer, and the prefill-decode split. The integration shape tells the story, the engines call into the Transfer Engine and Store for disaggregated prefill and cache reuse while retaining their own schedulers. For operators this means adoption is incremental: run vLLM exactly as today, add the Transfer Engine for a specific cross-node pain, then move toward full disaggregation if the workload justifies it. The relationship also future-proofs skills, because learning the caching and transfer concepts here transfers to any engine that integrates the layer, rather than binding you to one engine’s internals.

What does the KV cache actually cost, in numbers worth remembering?

A few anchors make the economics tangible. For a 70-billion-parameter model, the cache produced by a 128-thousand-token context runs about forty gigabytes, the exact figure the Transfer Engine benchmarks cite, which means one long conversation’s context rivals the storage footprint of the model’s own weights. At decode time, every generated token reads that entire cache from memory, which is why decode is bandwidth-bound and why cache locality dominates token latency. And across a serving fleet, prompt prefixes repeat heavily in real traffic, conversation turns, shared instructions, agent loops, which is the empirical fact that makes pooling valuable rather than theoretical. Hold those three anchors together and the design stops being exotic: forty-gigabyte states that must move fast, bandwidth-bound reads that punish distance, and repetition that rewards sharing. Any architecture change in serving is now judged against that triangle.

What should teams watch as this layer evolves?

Three threads. First, engine convergence: with both vLLM and SGLang integrating the Transfer Engine, disaggregated prefill is becoming a standard deployment shape rather than an exotic one, and the skills that matter shift from engine tuning toward cache-policy and network tuning. Second, the MoE connection: elastic expert serving is in the same repository because disaggregation logic generalizes from KV cache to expert activations, and mixture-of-experts models are becoming the default, so the layer underneath serving is being built for the model shape that is arriving rather than the one that is leaving. Third, the memory hierarchy keeps deepening, with CXL-attached memory and faster SSDs adding tiers below DRAM, and each new tier extends how long cache can live off-GPU. The through-line for planning: investments in understanding cache economics and RDMA networking age well across all three threads, while tight bindings to any single engine’s scheduling internals age fastest.

How would you actually evaluate Mooncake against your current stack?

With a workload replay rather than a synthetic benchmark, because the win comes from traffic shape and synthetics smooth traffic shape away by design. The method: capture a representative window of real requests, prompts, lengths, session structure, arrival times, then replay it against your current deployment and against a disaggregated one, measuring latency distributions per phase, cache hit rates on the pool, and cost per million tokens including the reclaimed GPU capacity. Three decision points fall out. If prefill-decode interference dominates your latency misses, the phase split alone justifies the work. If prefix repetition is high but your nodes are few, start with the Store for cross-node reuse before full disaggregation. And if your network cannot sustain RDMA-class transfer, fix the fabric first, because every other component inherits its ceiling. Teams that skip the replay and deploy on benchmark enthusiasm tend to rediscover their own traffic shape in production, expensively.

Where does the name come from and why does that matter?

Moonshot AI named its cache architecture after a filled pastry, and the whimsy is doing quiet work: it signals that the layer is infrastructure plumbing rather than a model or a product, a thing you compose into a stack rather than a thing you compare as a headline. Naming matters operationally too, because the ecosystem now references Mooncake components precisely, the Transfer Engine here, the Store there, as integration surface rather than as a monolith, and precise component names are what let two engines adopt different subsets without forking anything. When you read future serving-stack announcements, the vocabulary to watch for is exactly this granular: transfer engines, cache stores, pools, composable pieces adopted as needed.

What changed between the original paper and the current repository?

The 2024 technical report described the architecture; the FAST 2025 paper hardened it with production evidence and released the traces; and the repository since has decomposed the monolith into independently adoptable components, with the Store, the Transfer Engine, and the elastic MoE serving pieces opening on a staggered schedule the updates log records. The trajectory to watch in that sequence is componentization, each piece becoming usable without the others, which is how infrastructure layers mature from research artifacts into ecosystem staples. The released traces deserve particular credit, because reproducibility is the difference between an architecture people cite and one people deploy, and the team chose to be deployable.

The arc our inference coverage has tracked all year lands here: the KV cache went from an implementation detail to the asset around which serving architecture is organized. Mooncake is the production proof, award-stamped and engine-integrated, and the ideas it normalizes, phase separation, pooled cache, transfer-first design, are the ones to watch for in every serving announcement that follows. When the next engine or vendor talks about handling agentic context at scale, listen for whether the cache is a pool with a transfer budget, or still an accident of GPU memory. That distinction is the dividing line between the architectures built for this workload and the ones retrofitting to it For engineers, that listening skill is career-relevant rather than academic: the people who can hear whether a serving pitch understands its own cache economics are the people who end up owning the capacity plan when the pitch is purchased..