How Cloudflare Cut Its Inference Cost by 68%

Cloudflare routes every merge request through an AI code reviewer built from six specialist Claude agents, and by late August 2026 that pipeline billed $2.46 per review. The team spent September attacking the inference cost from four directions and got the same job done for $0.77, a 68 percent cut that Ryan Skidmore documented on the Cloudflare blog on October 19. The playbook transfers to any team paying per-token API bills.

The unit economics of one AI reviewer

The reviewer’s unit of account is one merge request. In April 2026 a review cost $1.19 on average. By the end of August the same pipeline billed $2.46, and the worst two-week window averaged $2.43. Nothing was visibly broken; the bill simply compounded.

Month in 2026	Average cost per review
March	$1.22
April	$1.19
June	$1.44
July	$1.74
August	$2.32, peaking at $2.46

Three forces stacked up. Moving all reviewers onto Anthropic models in early August took the per-job cost from $1.38 to $2.03. The standards reviewer’s move to Claude Opus 4.7 raised its slice from $0.25 to $0.88 per review. Tokens per review more than tripled over the same period, from 0.91 million in April to 2.99 million in September, while the overall cache hit rate actually improved from 85.7 to 97 percent.

The token profile behind most of that inference cost growth sat in the code quality reviewer. Before optimization it averaged 16.6 model calls per review, wrote 72,000 cache tokens, and read 1.08 million cached tokens per review. A single reviewer was moving a million tokens of context per merge request.

The spend split shows where the leverage was. Before optimization, cache writes made up 43 percent of spend and cache reads 34 percent, so input-side caching was 77 cents of every dollar. Output tokens, the part everyone watches, were 23 percent. Any serious attack on inference cost starts at the cached prefix.

One coordinator, six specialists, one shared prefix

Each merge request hits a coordinator that assigns one of three risk tiers: full review, lite, or trivial. A full review fans out to specialist reviewers for code quality, security, performance, AGENTS.md compliance, documentation, and standards. Each specialist runs as its own agent session against the Anthropic API, orchestrated by the open-source OpenCode agent.

Anthropic prompt caching only caches a prefix, laid out in a fixed order: tools, then system blocks, then messages, with at most four breakpoints per request. Cloudflare’s reviewers open with a repository-agnostic block of 11,000 to 21,000 tokens, byte-identical across repos. Keeping it identical took work: the skills list had to be disabled and tool descriptions frozen, because a cache hit requires byte-exact matches. Below the shared prefix sits the review-specific payload, the patch, the changed files, and the instructions for that specialist.

The diagram above shows the shape: one merge request becomes several sessions, and every session’s first request reuses the same cached opening. The inference cost of the whole system hinges on where the breakpoints sit.

The optimization ladder, ranked by what each rung saved

Cloudflare tried these fixes chronologically: cache, call cuts, then models. Ranked by what each saved, the order flips, and the flipped order is how a copycat should think about them. Together the rungs account for the full $1.67 drop in inference cost, from the $2.43 peak to $0.77.

Rung one: route each reviewer to the cheapest capable model, worth $1.06

Model and effort changes produced the biggest share of the saving, and they started from a simple observation: newer models at the same or lower effort level produced indistinguishable review outcomes. The coordinator dropped to a single model and effort combination, which made full-tier jobs about 20 percent faster. Claude Opus 5.5 at medium effort replaced Opus 5 with no measurable outcome difference.

A split test on October 3 compared Sonnet 5.5 against Sonnet 5 at identical per-token prices: code quality came out 62 percent cheaper, performance 58, security 47, and every point of it came from generating fewer tokens rather than a lower rate. The fleet moved to 5.5 on October 5.

Effort level needed its own measurement. A code quality review that had averaged 6.2 model calls dropped to 1.7 at medium effort, with more than half finishing in a single call, against 3.4 at high. But high effort on 5.5 posts 58 percent more critical and warning findings while costing 61 percent more per run, a premium of about two cents per review, so code quality went back to high.

Rung two: make the shared prefix cacheable, worth $0.45

The original prompt layout put environment details and the date inside the first system block, and the only cache breakpoint sat at the very end of the prompt. A cache key covers everything before its breakpoint, so a prefix that starts with bytes changing per repository and per day can never produce a shared hit. First-request hit rates ran between 14 and 26 percent even though 11,000-plus tokens of the opening were identical across every repo.

The September 16 fix split the block at a marker and placed a breakpoint right after the shared content. First-request cache hit rates jumped to between 91 and 99.5 percent. The same pass moved the standards reviewer’s reference material, its standards table, skill file, and reference list, into the prompt, cutting its calls per review from 15.9 to 9.6 and its cost from $1.12 to $0.72.

The reviewer did not get smarter; it stopped re-reading its own reference material at full price. Together these changes bent the inference cost curve from $2.47 to $2.02 per review.

How does Anthropic prompt caching price out?

Caching is billed as a multiplier on the base input token price, per Anthropic’s prompt caching documentation. These multipliers decide the inference cost math behind every fix in this piece:

Line item	Multiplier vs base input price
5-minute cache write	1.25x
1-hour cache write	2x
Cache read	0.1x list price, 0.05x on Opus 5.5 and Sonnet 5.5

The five-minute TTL refreshes at no cost every time the cached content is read, and the clock starts at request start, so generation time counts against it. Breakpoints are prefix-only in a fixed order of tools, system, then messages, with at most four per request, and a hit requires 100 percent identical preceding bytes. The minimum cacheable prompt on the 5.5 family is 512 tokens, against 2,048 on Opus 4.7.

Mapped onto the Cloudflare prompt, the walk after the September fix is: the frozen tool definitions, then the shared system block with a breakpoint right after it, then the repository context and date, then the review payload. A second reviewer arriving minutes later re-reads the first two segments at the read rate instead of full input price.

Two invalidation rules bit Cloudflare directly. Any change to a tool definition invalidates the entire cache, which is why tool descriptions had to be frozen. And changing the thinking or effort configuration invalidates the messages, which matters when you start testing effort levels per reviewer. That is why the September 16 fix was structural rather than a wording tweak: it moved the cacheable boundary to where the bytes stop changing.

Rung three: stop paying for dead work, worth about $0.20

Between September 21 and 23 the team stripped out work that changed no verdicts. Each move was small; together they ran on every job.

Move	What it fixed	Result
Removed unusable tools	in 57 to 76 percent of sub-reviewer sessions, a call was burned trying to invoke them	about 2,900 fewer tokens per request
Parallel file reads	64 percent of turns made exactly one tool call	first response reads the shared context and all patches, up to 150 KB
Skip reviewers on re-reviews	about two thirds of jobs are re-reviews	AGENTS.md skipped 56 percent of the time, performance 23, docs 13
Stop after posting	a final summary turn nobody read	about $0.07 per job
Cancel cancelled jobs	358 of 446 sampled cancelled jobs ran to completion	runner now polls status every 15 seconds
One first request for three reviewers	code quality, security, and performance share identical openings	reviewers two and three wait up to 20 seconds for the first write, saving about $0.09 each
Separate slow from broken	security timeouts ran at 1.8 percent	timeouts now near zero

The cancellation row hides the biggest single waste. The runner sits on GitLab with a Kubernetes executor, which never signals cancellation, so a developer who hit cancel still paid for the full review until the runner started polling status.

What the inference cost cuts did to speed and quality

Cheaper was also faster. The median review went from 4 minutes 9 seconds to 1 minute 32 seconds, and the 99th percentile from 15 minutes 20 seconds to 4 minutes 41 seconds. The share of reviews running past five minutes collapsed from 37.9 percent to 0.7 percent, which changes what a reviewer is for: at five minutes you wait for it, at 90 seconds you read it.

Review tier	Median before	Median after
Full	5 min 16 s	1 min 54 s
Lite	2 min 58 s	1 min 14 s
Trivial	77 s	30 s

Quality moved in both directions, and the direction was measured rather than assumed. The act-on rate for code quality findings, the share where the author changed the flagged lines in the next push, rose from 34 to 59 percent. Acted-on findings per 1,000 reviews rose from 229 to 321 across all reviewers, even as code quality’s own count dipped from 145 to 132. Severity over-rating fell 70 percent, from 0.46 to 0.14 over-rated findings per review. Critical and warning findings dropped 40 percent on code quality, 56 on security, and 71 on performance.

The harness that made the cuts safe

None of those steps down would have been safe to take blind. Cloudflare measured review quality two ways, and both are copyable. The first counted whether the author acted on a finding: if the flagged lines changed in the next push, the finding counted as acted on. To calibrate the bar, the team also sampled lines 30 to 40 positions away from any finding, and about 15 percent of those untouched lines changed anyway. That coincidence floor is what an honest act-on rate has to clear, and 59 percent clears it comfortably.

The second method graded a blinded sample of 657 findings with models from two separate vendors, with a third model breaking ties. Each finding received a label of important, minor, duplicate, wrong, or unverifiable, plus a severity check of fair, too high, or too low. Sample sizes ran 35 to 66 findings per group, and the final comparison window was a single day, so the deltas are directional rather than definitive.

Step comparisons also use weekday hours before and after each change rather than a controlled experiment, and Cloudflare says so plainly. That honesty surfaced the real losses: the performance reviewer shed about 64 percent of its important findings, roughly one real finding per 24 full reviews, and medium-effort code quality lost about three quarters of its acted-on findings compared with high. Security fell about 25 percent, within noise.

Reliability moved the other way for everyone. Security retries fell from 3.5 to 0.04 percent and reviews that ended without a verdict from 0.79 to 0.27 percent. The platform keeps moving too: OpenCode v2, a ground-up rewrite, matched v1’s approval decisions 98 percent of the time in shadow mode with full-tier reviews about 13 percent faster, and took over in early October. Next on the experiment list is running reviewers on Cloudflare Durable Objects via the pi-durable runtime.

How did Cloudflare tell real findings from noise?

The act-on method has one more refinement worth knowing. Because roughly 15 percent of untouched lines change anyway, a reviewer that flags busy code can rack up acted-on findings by accident. Comparing the act-on rate against that baseline, rather than against zero, is what separates signal from churn, and it needs nothing more than a diff and a push timestamp.

The blind grading added the texture the act-on rate cannot see: which findings were duplicates, which were wrong, and whether severity labels were inflated. On those terms, important findings per review held between 0.28 and 0.30 while over-rated severity fell from 0.46 to 0.14 per review. Critical and warning findings fell for the three Sonnet reviewers while the compliance reviewer posted more, yet acted-on findings per review rose, which reads as fewer, better-placed comments rather than a quieter reviewer. Cloudflare’s parting advice belongs on this list: confirm that the first request of every session is served from cache rather than fresh input, and that the reviewers still surface real defects at the effort tier you are buying.

Which reviewers lost ground as the cost fell?

The performance reviewer took the real hit: 71 percent fewer critical and warning findings, and about 64 percent fewer important ones, which works out to one lost real finding per 24 full reviews. Security findings fell about 25 percent, a delta Cloudflare reports as not statistically significant. Code quality at medium effort lost roughly three quarters of its acted-on findings against high effort, which is why that reviewer moved back up.

Even so, the Sonnet 5.5 switch had already cut code quality’s cost 62 percent before the step up to high effort added back about two cents per review. None of these deltas were visible in the topline cost numbers; they came out of the per-reviewer breakdown, which is the level anyone copying this playbook has to measure at. September’s token counts frame the whole ladder: cache reads down 82 percent, cache writes down 54 percent, output tokens down 67 percent.

Do these tricks transfer to a self-hosted vLLM stack?

The inference cost multipliers do not transfer, because a self-hosted cluster rents GPU-seconds instead of tokens. The prefix discipline does. vLLM’s automatic prefix caching, enabled with enable_prefix_caching=True, reuses the KV cache of shared prompt prefixes across requests, so a fleet of agents opening with the same 11,000-token system block pays the prefill compute once instead of per session. The vLLM documentation names exactly the two patterns Cloudflare exploited: repeated queries against the same long document, and multi-round conversations.

The same limits apply on both sides. Prefix caching only speeds prefill, so decode-heavy workloads with long answers see little benefit, and requests that share no prefix see none at all. That last condition is the self-hosted restatement of Cloudflare’s broken breakpoint: identical bytes in, reused compute out. The call-count and cancellation moves transfer without modification, because an agent that burns a turn on an unusable tool inflates inference cost on rented GPUs just as it does on a per-token API. The measurement discipline travels best of all: asking whether prefixes are being reused and whether findings stay real costs nothing on any stack.

The system now optimizes for token economics: cache every byte that repeats, delete every call that changes no verdict, and buy model power only where a measured quality delta justifies the price. Inference cost fell 68 percent, from $2.43 to $0.77 per review, while reviews got faster and findings got more precise. The tradeoff is engineering attention, because the measurement harness took more effort than any single fix, and without it the same cuts would have silently shipped worse reviews.