30 sec3 min coredeep dive optional
Query-aware compression cuts RAG costs on Amazon Bedrock by 33 percent in AWS’s own benchmark: one extra small-model call strips retrieved context down to the spans that answer the actual query before the expensive model ever sees it.
- Tokens reaching the primary model dropped to 12 percent of baseline, an 8.6x reduction, with answer quality at 97.5 percent of baseline.
- The compression step is verbatim span extraction at temperature 0.0, never summarization, so citations and figures survive intact.
- The tradeoff is latency: 12 to 19 percent slower end to end, which rules the pattern out for sub-second chat surfaces.
- Published by AWS ProServe on August 21, 2026, from a benchmark of 500,000-plus documents, 500 questions, and 10 query categories.
RAG cost problems hide in plain sight. Your retriever pulls five to twenty chunks because you asked it to, and every one of those tokens bills on every query, whether or not they contain the answer. Query-aware compression attacks that bill directly: a small model reads the retrieved chunks against the query, copies out only the spans that matter, and the large model answers from a fraction of the context. AWS published the pattern and its benchmark numbers on August 21, 2026, and the results are strong enough to prototype this week.
What query-aware compression changes in a RAG pipeline
The baseline flow is familiar: embed the query, fetch top-k chunks, concatenate everything into a prompt, and make one expensive call. Compression inserts exactly one new step between retrieval and the answer call. A small model, priced far below the primary, receives the query plus all retrieved chunks and returns the query-relevant spans verbatim. The primary model then answers from that trimmed evidence instead of the full dump.
The economics reduce to two numbers. The first is the price ratio between your small and large model. The second is the compression ratio you actually achieve, which depends on how much of your retrieved context is dead weight. Multiply the small model’s added cost against the large model’s savings and you get the sign of the result. In AWS’s benchmark the small model’s cost was noise next to the savings: total cost fell to 67 percent of baseline with compression alone, and to 64 percent when paired with reranking.
One design decision carries most of the value: the small model extracts spans, it does not summarize. Verbatim extraction keeps source wording intact, which is what citations, exact figures, and regulated-industry audits all require. A summary layer would paraphrase away the very thing you are paying for.
The pipeline: one Lambda, two model calls
AWS’s reference shape puts both calls inside a single Lambda function using the Bedrock Converse API, with model IDs read from environment variables. Retrieval stays upstream: Amazon Bedrock Knowledge Bases, backed by OpenSearch Serverless, is the named compatible retriever. The application passes the query plus all top-k chunks to the Lambda; the Lambda calls Claude Haiku for compression, then Claude Sonnet for the answer. Suggested pairings keep both models in one family: Haiku with Sonnet, or Amazon Nova Micro with Nova Pro.
Here is a minimal implementation of that shape, written from the documented mechanics:
import boto3, os
bedrock = boto3.client("bedrock-runtime")
COMPRESS_MODEL = os.environ["COMPRESS_MODEL_ID"] # small model, e.g. a Haiku ID
ANSWER_MODEL = os.environ["ANSWER_MODEL_ID"] # primary model, e.g. a Sonnet ID
def handler(event, context):
query = event["query"]
chunks = event["chunks"] # [{"id": "c1", "text": "..."} ...]
evidence = compress(query, chunks)
return {"answer": answer(query, evidence)}
def compress(query, chunks):
corpus = "\n\n".join(f"[{c['id']}]\n{c['text']}" for c in chunks)
system = (
"Copy verbatim every span of the corpus that answers the question. "
"Never summarize or paraphrase. Keep each span's [chunk id] tag. "
"If a chunk holds no relevant evidence, return its id with "
"NO_RELEVANT_EVIDENCE."
)
out = bedrock.converse(
modelId=COMPRESS_MODEL,
system=[{"text": system}],
messages=[{"role": "user", "content": [
{"text": f"Question: {query}\n\nCorpus:\n{corpus}"}
]}],
inferenceConfig={"temperature": 0.0},
)
return out["output"]["message"]["content"][0]["text"]
def answer(query, evidence):
out = bedrock.converse(
modelId=ANSWER_MODEL,
system=[{"text": "Answer only from the evidence. Cite chunk ids."}],
messages=[{"role": "user", "content": [
{"text": f"Question: {query}\n\nEvidence:\n{evidence}"}
]}],
inferenceConfig={"temperature": 0.0},
)
return out["output"]["message"]["content"][0]["text"]Temperature 0.0 on the compression call is not a style choice; it is what makes extraction deterministic enough to trust and to test. The answer call reads a system instruction that restricts it to the provided evidence and requires citations, which is standard grounding discipline applied to a much smaller evidence block.
What the span-extraction prompt must enforce
The compression prompt is the load-bearing part of the whole pattern, and AWS’s version states four constraints worth copying exactly.
| Constraint | Why it matters |
|---|---|
| Extract verbatim spans, never summaries | Paraphrase corrupts figures, names, and citation text |
| Keep surrounding context around each span | Sentences lose meaning when yanked out of their neighborhood |
| Carry chunk IDs through to the output | Citations in the final answer must trace back to sources |
| Mark empty chunks with NO_RELEVANT_EVIDENCE | Silently dropping chunks makes compression quality unauditable |
AWS also flags that the posted prompts are examples, not drop-in. Every domain needs its own adapted compression prompt, and the benchmark’s quality numbers came from prompts tuned for its corpus. Budget an iteration loop for this prompt the same way you would for a reranker’s cutoff thresholds.
The benchmark numbers, honestly read
AWS ran the comparison on a corpus of more than 500,000 documents across nine enterprise source types, with 500 questions across 10 categories, scored by an LLM judge on four dimensions plus a separate faithfulness check. Three conditions: baseline, compression, and rerank plus compression.
| What changed | Uncompressed start | After compression | Compression with reranking |
|---|---|---|---|
| Tokens reaching the primary model | 100 | 12 (8.6x fewer) | 10 (10.1x fewer) |
| Total cost (indexed) | 100 | 67 | 64 |
| Answers containing unsupported claims | 51% | 44% | 38% |
| Answer quality composite | 100 | 97.5 | 97.6 |
| End-to-end latency overhead | none | +19% | +12% |
Two rows deserve a second look. The hallucination row moves the most, dropping 7 points with compression and 13 with reranking, because irrelevant context never reaches the model that generates claims. The latency row is the honest cost: the extra call adds a few hundred milliseconds to about a second, partially recovered because the primary model reads less. AWS repeats a caveat three times in the post, and it deserves the repetition: every figure above was measured inside a single testbed, on one body of documents, one subject area, and one style of question. Your numbers will differ, which is why the fit question matters more than the headline.
When query-aware compression pays for itself
The fit heuristic is refreshingly concrete. Prototype the pattern when retrieved context averages above roughly 5,000 tokens, when your questions are narrow relative to what you retrieve, and when a 3 to 10x context cut is worth real money at your volume. Skip it when any of those fail: latency-critical chat over small contexts loses more to the extra call than it gains in tokens, and AWS says so explicitly, pointing those workloads at prompt caching or Intelligent Prompt Routing instead.
The winning workloads share a shape. Compliance and policy assistants, support copilots, runbook and wiki assistants, and financial research over 10-K sections or earnings transcripts all retrieve fat, mostly-irrelevant context to answer narrow questions. That is the compression profile. Hard multi-part queries save less everywhere: savings fell from 37 percent on the typical set to 26 percent on the hard set, and from 40 to 30 percent with reranking.
Finally, this stacks rather than competes with Bedrock’s native cost levers. Prompt caching cuts the repeated-prefix bill, Intelligent Prompt Routing sends easy queries to cheaper models, the Rerank API tightens what compression receives, and Guardrails validate grounding on the way out. Compression composes with all four because it operates on a different axis: how many tokens the answer model reads at all.
Ship it: prerequisites and rollout checks
The setup list is short. You need model access enabled for both models in your Region, an IAM role granting Lambda the Bedrock permissions, and a retriever feeding chunk IDs alongside text. From there, rollout is a measurement discipline:
Why verbatim spans instead of a summarizing compressor?
Summarization is the obvious idea and the wrong one, for three compounding reasons. First, citations: the answer model cites sources by pointing at evidence text, and a paraphrased citation no longer matches the stored document, which breaks both human audit and automated citation-accuracy scoring. Second, figures: summaries routinely round, merge, or drop numbers, and in the regulated domains where this pattern fits best, the numbers are the answer. Third, trust: a verbatim span can be diffed against its source in one line of tooling, while a summary requires a model to judge it, which reintroduces exactly the error surface you were paying to remove.
The verbatim rule also explains the temperature setting. Extraction is a search problem, not a generation problem: given the same query and corpus, the same spans should come back every time. Temperature 0.0 plus a strict no-paraphrase instruction moves the compression call from “probably similar” to “reproducible,” which is what lets you write tests against it. A nondeterministic step inside your cost-critical path is a paging incident waiting for a quiet Friday.
There is a quieter benefit in the benchmark’s faithfulness methodology. Claims were checked against the evidence the primary model actually received, and the hallucination rate fell from 51 to 44 percent, and to 38 with reranking, precisely because the evidence shrank while staying faithful to sources. A summarizing layer would have optimized the same metric the other way: less evidence, but evidence you cannot vouch for, which shows up as confident answers grounded in nothing checkable.
The cost of the rule is real and worth naming: verbatim spans compress less than summaries would. You give up a few points of compression ratio in exchange for auditability. Given the measured 8.6x, the trade is good. If your corpus ever pushes you below the 3x band where savings get thin, the correct next step is tightening retrieval or adding rerank, not loosening the verbatim rule.
A small tooling investment locks the rule in. A nightly job that samples compressed outputs, checks each span against its source chunk, and alerts on the first paraphrase costs an afternoon to build and catches prompt drift before your citation-accuracy metrics do.
How do the economics actually work out per million queries?
Walk the arithmetic. Every compressed query now pays for two model calls instead of one: the small compressor reads the full retrieved context, and the primary model reads the compressed output. The savings live entirely in the second call’s input tokens, which drop to roughly 12 percent of baseline. Whether the pattern wins depends on your small-to-large price ratio. At a ratio of about one to twelve, the compressor’s read of the full context costs about as much as the primary model’s read of the compressed context, and the win comes from everything beyond tokens: output pricing, request overhead, and extra headroom on provisioned throughput.
The benchmark’s own numbers are the cleaner illustration. Total cost fell to 67 percent of baseline with compression and 64 percent with rerank plus compression, while tokens to the primary model fell to 12 and 10 percent. The gap between those two facts is the compressor’s own bill plus what the primary model still reads. Hard queries compress the ratio toward breakeven, saving 26 to 30 percent instead of 33 to 36 percent, because multi-part questions keep more of the retrieved context relevant.
Latency bills differently from dollars but lands on the same spreadsheet. The added call costs a few hundred milliseconds to about a second, measured at 12 to 19 percent overhead end to end. Part of that returns because the primary model reads less, and the rerank pairing actually measured faster than compression alone at +12 percent, likely because reranking shrinks the compressor’s input too. For internal tools and research assistants this is invisible. For customer-facing chat with a latency SLO, it is the deciding row.
To model your own case, you need one number AWS cannot give you: your corpus’s compression ratio. Get it cheaply. Pull fifty real queries from your logs, run the compressor on their actual retrieved context, and measure the span output size against the chunk input size. Multiply that ratio against your average retrieved-context tokens and you have the primary model’s new input bill; price the compressor’s read at your small model’s input rate; add both to the unchanged output pricing and request overhead.
If the modeled saving lands under about 15 percent, spend the engineering time on prompt caching or retrieval tuning first and revisit when context sizes grow, because the compression prompt’s iteration cost will not pay for itself at that margin.
Two correction factors keep the model honest. First, query mix: the benchmark’s hard-query column is the warning label, since multi-part questions hold their savings 10 to 11 points below the typical set, and a workload heavy on broad research questions should use the hard column as its planning number. Second, staleness: retrieval distributions drift as the corpus grows, so a compression ratio measured at launch silently decays, and the fifty-query measurement is worth rerunning quarterly alongside the rest of the cost review.
What breaks when the compressor drops evidence it should have kept?
The failure surface moves. In baseline RAG, the answer model sees everything and hallucinates around irrelevant context; with compression, the answer model is accurate but blind to whatever the compressor discarded. The benchmark’s quality metrics capture the residue: composite scores held at 97.5 percent of baseline, but completeness and citation accuracy dipped slightly while conciseness improved, which is the signature of evidence getting trimmed a little too eagerly on answers that needed breadth.
Three defenses keep the failure contained. First, the NO_RELEVANT_EVIDENCE convention: chunks that the compressor judged irrelevant are marked, not deleted, so you can audit rejections and spot systematic misses. Second, a small eval set scored on the same dimensions AWS used; if completeness degrades beyond your threshold, the compression prompt is too aggressive for that query class. Third, post-hoc span validation, which the post names as an optional safeguard: a cheap check that every extracted span actually appears verbatim in its source chunk, catching paraphrase drift the moment the prompt starts to slip.
Query type predicts failure mode. Narrow factual lookups compress safely because relevance is unambiguous. Broad multi-part questions are where over-compression bites, and they are also where savings shrink toward breakeven, which suggests a clean routing rule: compress narrowly-scoped queries, let hard queries through uncompressed, and you concentrate the savings where risk is lowest. That routing decision costs one cheap classifier call and removes the worst tail of the risk.
Finally, keep the human audit path alive. Because spans are verbatim and carry chunk IDs, any answer can be traced in seconds: evidence span, source chunk, stored document. That traceability is the pattern’s real safety net, and it is why the verbatim discipline from the first deep dive is not just a quality preference. Drop it, and every downstream safeguard loses its ground truth.