A look at how ChatGPT works. ChatGPT served 100 million weekly active users within two months of launch, the fastest consumer-software adoption in history. The architecture that scaled to that size wasn’t new technology; it was the careful application of patterns every ML inference system has used for years, plus one trick: the per-session KV-cache that makes multi-turn conversations cheap.
How ChatGPT Works: The problem at scale
LLM inference is expensive and slow. Generating one token from GPT-4-class models takes 30-100ms on the best available GPU, and a typical response is 200-500 tokens: that’s 6 to 50 seconds of compute per response, per user, per turn, with no economies of scale across users because every conversation is unique. ChatGPT’s challenge: serve hundreds of millions of conversations per day, each one streaming tokens back to the user in real time, while keeping cost-per-conversation low enough to make the product economically viable.
The core architecture
When you send a message, the request hits the API Gateway (auth, rate-limit), then the Orchestrator. The orchestrator is the brain: it loads your conversation history from Postgres (long-term) + Redis (active-session cache), runs tools (web browse, code execution, file parsing) if needed, retrieves RAG context for cited answers, and assembles the prompt. It then calls the Router, which picks the model (GPT-4o, GPT-4o-mini, o1, etc.) based on the request type and user plan. The router sends the prompt to the Inference Cluster (thousands of H100/H200 GPUs), which streams tokens back through the orchestrator to the client, while a Safety Classifier runs in parallel pre- and post-generation to filter disallowed content.
The KV-cache is what makes multi-turn work
An LLM is fundamentally an autoregressive transformer: each new token requires recomputing attention over all prior tokens. For a 4K-token conversation, that’s 4K × N matrix multiplies per token, billions of FLOPs. The trick that makes it affordable: cache the attention key-value tensors (KV-cache) from prior turns. On the next turn, only the new tokens need their KV computed; prior turns’ KV tensors are loaded from the cache. This drops per-turn compute by 10x-100x for long conversations, depending on context length.
OpenAI’s not unique in doing this (every serious inference provider does), but the engineering is in keeping the cache warm per active session, on the right GPU, without paging it to host memory. Sessions that go idle get evicted; resuming them requires recomputing KV from the conversation history, which is why your first message after coming back from lunch feels slower than the next one.
The router is the cost-control layer
Not every query needs GPT-4o’s intelligence. The router decides which model handles each request based on intent classification, complexity signals, and the user’s plan (free, Plus, Pro). A “what’s the time in Tokyo” question goes to GPT-4o-mini (cheap, fast); a “debug this Python async race condition” question goes to GPT-4o or o1. This is the lever that makes the free tier economically viable: most queries are simple enough to route to the small model. The router is itself a model, continuously retrained on routing decisions vs response quality.
The one non-obvious insight
Streaming isn’t a UX feature; it’s an architectural necessity. If ChatGPT waited to return the full response, the user would stare at a spinner for 20-40 seconds before seeing anything. By streaming tokens as they’re generated, the perceived latency drops to first-token (~1 second) and the user reads along as the model writes. The same total compute happens either way; streaming just shifts the perceived wait. This is why every chat-style LLM product streams: not for polish, but because the alternative is unusable.
ChatGPT is the canonical LLM-serving stack: API gateway → orchestrator with tools/RAG → router → GPU cluster with KV-cache → safety classifier. The architecture isn’t exotic, but the operational engineering to keep KV-caches warm and the router honest is what made 100M users possible in 60 days.
Continue learning: