5 AI Infrastructure Projects to Build (2026)

Every token a model processes produces keys and values that live in GPU memory as the KV cache. When the next request from the same conversation lands on the same engine, all of that prefix work is reused and the model starts generating almost immediately. Land it on a different engine, and the prefix gets recomputed from scratch.

A naive round-robin load balancer ignores this completely: it scatters sessions across replicas and throws the reuse away. The production-stack router instead directs requests by session ID and routing key so a conversation tends to stay with the engine that already holds its cache, and prefix-aware matching extends the same idea to shared system prompts. LMCache can additionally offload cache to CPU memory so an engine survives more concurrent context than its GPU alone could hold.

You can watch the difference instead of taking it on faith. The stack’s Grafana dashboards surface per-engine metrics, so a routing change shows up in the numbers within minutes: cache hit rate, requests per engine, time-to-first-token. Run the same conversation twice, once against round-robin and once against session routing, and compare the second request’s latency in each setup. That single measurement explains why every serious serving platform grew a router.

The payoff shows up as lower time-to-first-token and higher throughput per GPU, which is why routing sits at the center of this stack rather than in front of it as an afterthought. We trace the full path of a request, router included, in How ChatGPT Works: KV-Cache, GPU Cluster, and the Router, and the serving engines roundup explains where vLLM fits among its alternatives.

Kubernetes autoscaling is three mechanisms stacked, not one. The Horizontal Pod Autoscaler adds or removes pods based on metrics it scrapes from those same pods: CPU, memory, or a custom metric an exporter publishes. It is the default answer, and it cannot scale to zero, because once the last pod is gone there is nothing left to report metrics.

KEDA closes that gap by watching an external signal instead: a Redis queue length, a Kafka consumer lag, a Prometheus query. Because the signal lives outside the workload, KEDA can hold a deployment at zero replicas and wake it on the first event. The Cluster Autoscaler works one layer down, on nodes: it notices pods stuck pending for lack of capacity, asks the cloud provider for a machine that fits, and removes nodes that stay underutilized.

Each layer also has its own timing, and the timing is where tuning lives. The Horizontal Pod Autoscaler evaluates on a sync interval and applies a stabilization window before scaling down, so a traffic spike that ends in ninety seconds may never trigger a removal at all. KEDA adds its own cooldown and stabilization settings per scaler, and the Cluster Autoscaler waits out a node’s underutilization before deleting it. Stack those delays deliberately: aggressive scale-down paired with slow scale-up produces a system that oscillates, and the cure is almost always more patience on the way down.

The KEDA repo in the last project chains all three deliberately, which is why its architecture is worth reading even if you never run the GPU phase: queue depth wakes pods, pending pods wake nodes, and quiet tears it all back down.

Walk the timeline of the first request after an idle period. It lands in Redis and waits. KEDA polls the queue on its configured interval, notices the depth, and creates pods. If no node can fit a pod that requests a GPU, the Cluster Autoscaler provisions a VM, which then boots, joins the cluster, and gets its NVIDIA drivers and container runtime in order. Only after that does the image pull finish, the model weights load from the persistent volume, and vLLM open its port. The request that started all of this finally gets answered.

Measure the chain before you judge it. Timestamp the request when it enters Redis, when KEDA creates the pods, when each pod becomes ready, and when the first token comes back. The gaps between those stamps tell you which step owns your cold start, and it is rarely the one you suspect: node join usually dominates on a real GPU cluster, while image pull and weight loading dominate once nodes are warm. Continuous batching helps on the recovery side, because once vLLM is up, new requests join batches already in flight and throughput climbs quickly.

That chain is why the repo buffers through Redis at all: nothing is dropped and no client needs retry logic, but latency on the first request is the price. The repo’s own mitigations, weights on a PVC and pre-cached image layers, attack the two slowest steps directly. The honest conclusion is that scale-to-zero fits bursty and internal workloads, while anything with steady traffic should keep a minimum replica count and accept the bill. A useful compromise is a minimum of one replica during working hours and zero overnight, which keeps the worst cold start out of your busiest window.

AIOps applies pattern recognition to operational telemetry. Metrics, logs, and traces flow in; the system detects anomalies against learned baselines instead of fixed thresholds, correlates the resulting alerts so one root cause doesn’t page you twelve times, and then suggests or executes a response: restart a deployment, scale a service, roll back a release.

The telemetry these serving projects export is exactly what an AIOps layer consumes. Queue depth from Redis, time-to-first-token and KV cache state from vLLM’s exporter, GPU utilization and power draw from DCGM, pod and node lifecycle from kube-state-metrics: wire those into the platform and the anomaly detection has something real to detect. That is the cleanest way to see the relationship between the two halves of this list. The serving projects produce the signals, and the operations layer decides what to do when the signals go wrong.

Its limits matter as much as its promises. The detection layer is only as good as the telemetry feeding it, and the automation layer usually earns trust gradually, starting with actions a human approves before graduating to the ones nobody reviews. The useful mental model is a control loop wrapped around the same dashboards you already run, closing by itself for the boring failure modes first and escalating the rest. Start with alert correlation alone; it is the fastest win and the easiest part to evaluate.

Not all five carry the same weight. The Ollama Dockerfile generator is a warm-up, and listing it alone signals beginner. The vLLM production stack and the KEDA scale-to-zero build are the two that read as production engineering, because they involve the decisions teams actually pay for: routing, replica counts, node costs, cold starts. The KServe samples are a lab; fold them into a story about autoscaling rather than leading with them.

Whichever you pick, be ready to answer what broke. The credible line is not “deployed vLLM on Kubernetes” but something closer to “deployed a multi-replica vLLM stack and cut time-to-first-token by fixing session routing”, because the second version contains a measurement and a decision. Build one project deeply, break it on purpose, benchmark the before and after, and you have a better story than someone who skimmed all five.

A practical sequencing if you have one free weekend a month: build the Ollama tool first for confidence, spend the next month on the production stack, then do the KEDA build in a third sprint. By the last one you will recognize every component because you have already run it somewhere else, and that recognition is most of what the job title actually describes.