06 – AWS RAG Optimization: Writing for Retrieval Accuracy

RAG optimization begins before a user submits a query — at the source documents that feed retrieval-augmented generation. Document and context engineering is the discipline of writing for two readers: the human who skims the page and the embedding model that chunks it.

RAG Optimization: What You’ll Learn

RAG optimization is the practice of writing and structuring source content so retrieval surfaces the right passages. The work happens at the document layer — you shape the corpus the pipeline chunks, embeds, and ranks. This guide covers the document-level levers that drive accuracy: heading structure, paragraph shape, list depth, redundancy, pronoun removal, and non-text content handling.

Why Document Quality Determines RAG Accuracy

The single biggest determinant of retrieval accuracy is document quality. Amazon Bedrock Knowledge Bases chunks source documents into passages of approximately 300 tokens by default (configurable via chunking strategy); each chunk is embedded independently into a vector in Amazon OpenSearch Service. If a chunk is self-contained, the model gets clean context; if it depends on a heading three paragraphs up or a pronoun elsewhere, the model gets garbage.

A weak embedding model on well-optimized documents outperforms a state-of-the-art model on messy ones. The AWS Prescriptive Guidance frames this as context engineering — the deliberate shaping of the context the model receives.

How RAG Retrieval Works (Brief)

The full mechanics are in the AWS Prescriptive Guidance on writing best practices for retrieval-augmented generation. In short: Bedrock Knowledge Bases splits documents into chunks, runs each through an embedding model (such as Amazon Titan Text Embeddings v2), and writes the vector plus metadata into a vector store — OpenSearch Service, Aurora PostgreSQL with pgvector, or Kendra.

At query time the question is embedded the same way, the store runs a similarity search (typically cosine), and the top-K chunks are concatenated and sent to the LLM.

Bedrock Knowledge Bases exposes four chunking strategies — fixed-size, hierarchical, document-based, and semantic. Semantic and hierarchical reward well-structured documents and punish messy ones, relying on heading boundaries and topical coherence.

Each strategy takes different configuration parameters when you define a data source. Fixed-size chunking takes maxTokens and overlapPercentage — a common starting point is 300 tokens with 10–20% overlap so a sentence spanning a chunk boundary is not orphaned on either side.

Hierarchical chunking takes parent and child token sizes, for example a 1,500-token parent chunk holding several 300-token child chunks. The retriever searches on the child chunks for precision but returns the parent chunk to the model for fuller context, which reduces the “correct chunk, missing context” failure mode where the right passage was found but too narrowly clipped to answer the question.

Semantic chunking takes a breakpointPercentileThreshold and a bufferSize. It embeds sliding groups of sentences and cuts where the embedding distance between consecutive groups spikes, which is exactly why well-marked topic transitions — clear headings, one idea per paragraph — produce cleaner semantic chunk boundaries than a wall of undifferentiated prose.

Document-based chunking treats each file as one chunk and is only appropriate for short, already-atomic documents such as FAQ entries or short policy snippets. Using it on long multi-topic pages defeats the purpose of chunking entirely, since the whole page becomes a single, diluted embedding that matches nothing precisely.

Re-embedding a corpus after a chunking-strategy change is not free. Every existing vector in the OpenSearch Service index becomes stale and must be deleted and re-ingested, and any application code caching chunk IDs needs to treat the change as a breaking schema migration rather than a routine content update.

Test a new chunking strategy on a representative sample data source first, and compare recall@K against the current strategy on your evaluation set before touching production. Only re-point production data sources at the new configuration once it wins on held-out queries — a strategy that looks better in isolated spot checks can still regress overall recall if it happens to fragment a handful of high-traffic documents differently.

RAG Optimization: Document Structure Best Practices

Use clear, descriptive headings. Headings are the highest-signal element for a chunking pipeline; Bedrock Knowledge Bases uses heading boundaries to split. “Configure the IAM role” produces a chunk capturing that concept; “Step 3” produces one dominated by body prose.

Keep paragraphs short and self-contained. Three to five sentences survive chunking intact and embed as complete units. Treat every paragraph as a potential retrieval unit — if it were the only context the LLM got, would it still be coherent?

Prefer flat lists over nested ones. Lists three or more levels deep force the chunker to split between levels (orphaned sub-items) or embed the whole list (diluting the vector). If a list needs more than two levels of indentation, use a table or short prose.

Add a brief summary after each major section. A one- or two-sentence summary produces a chunk at the centroid of the section’s topic in vector space.

Writing Style That Improves Retrieval

Define abbreviations on first use. Write “Retrieval-Augmented Generation (RAG)” on first appearance — the chunk with the definition becomes the canonical retrieval target.

Avoid anaphora. Pronouns — “it”, “this”, “they”, “that” — are the largest avoidable noise source. “It processes the data and returns it in a format the system can use” embeds verbs but not subjects. Rewrite: “Amazon Bedrock Knowledge Bases processes the raw document chunks and returns structured embeddings that Amazon OpenSearch Service uses for similarity search.”

Use the active voice. “Lambda processes the request” places the actor first and produces a tighter embedding than passive voice.

Prefer concrete nouns. “Amazon Bedrock Knowledge Bases handles the document ingestion workflow” embeds with Bedrock-specific signal; “The system handles the workflow” embeds as a generic cluster.

Keep one idea per paragraph. Multi-topic paragraphs embed as a blend that retrieves at medium relevance for many queries but high relevance for none.

Managing Redundancy and Token Efficiency

Redundant content is the silent killer of retrieval accuracy. When the same information appears in five places, each copy chunks and embeds separately; at query time, three may surface in top-K, consuming budget that should go to unique information. The fix is one canonical statement per fact, cross-referenced rather than duplicated.

Build a duplicate-embedding detector: compute embeddings for all chunks, find pairs with cosine similarity above 0.95, and consolidate. OpenSearch Service supports k-NN search natively, making this cheap. Use a tiered threshold: consolidate above 0.95, flag 0.85 to 0.95 for review, leave below 0.85 alone.

In practice this is a self-join against the same k-NN index the Knowledge Base already populated — for every chunk vector, query the index for its nearest neighbors excluding itself, and act on any result above the threshold:

from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3

session = boto3.Session()
creds = session.get_credentials()
auth = AWS4Auth(creds.access_key, creds.secret_key, session.region_name, 'aoss', session_token=creds.token)

client = OpenSearch(
    hosts=[{'host': 'kb-index.us-east-1.aoss.amazonaws.com', 'port': 443}],
    http_auth=auth, use_ssl=True, verify_certs=True,
    connection_class=RequestsHttpConnection,
)

def find_near_duplicates(index, threshold=0.95, k=5):
    results = []
    hits = client.search(index=index, body={"size": 10000, "query": {"match_all": {}}})["hits"]["hits"]
    for hit in hits:
        vector = hit["_source"]["embedding"]
        neighbors = client.search(index=index, body={
            "size": k,
            "query": {"knn": {"embedding": {"vector": vector, "k": k}}},
        })["hits"]["hits"]
        for n in neighbors:
            if n["_id"] != hit["_id"] and n["_score"] >= threshold:
                results.append((hit["_id"], n["_id"], n["_score"]))
    return results

Run this monthly against each active data source rather than continuously. A full self-join over a large index is expensive, and redundancy tends to accumulate slowly as writers copy boilerplate between documents rather than appearing overnight.

Pairs above 0.95 are almost always a paragraph pasted verbatim into two documents; the fix is deleting one copy and replacing it with a short cross-reference sentence rather than leaving both to compete for the same query.

Pairs in the 0.85–0.95 band are usually two documents describing the same concept from different angles — often worth keeping both, but worth a human pass to confirm they are not simply stale duplicates of a since-updated procedure that never got cleaned up.

Handling Images, Tables, and Non-Text Content

Describe every image in text. Embedding models are text-only; an image produces no signal unless described. A caption like “Figure 3: The Bedrock Knowledge Bases ingestion pipeline — source documents enter from Amazon S3, are chunked, embedded by Titan Text Embeddings, and stored in OpenSearch Service” retrieves for ingestion, chunking, and vector storage queries.

Convert tables to prose where possible. Tabular data chunks badly — cell relationships are lost when the chunker splits mid-row. Small tables should become prose; large tables need a summary.

Use alt text on every image. Bedrock Knowledge Bases for HTML/web content extracts alt text during chunking, making it another retrieval surface.

Agentic RAG and Context Engineering

Agentic RAG layers reasoning on top of retrieval: a planner decomposes the query into sub-queries, each is embedded and run against a Bedrock Knowledge Base, a critic re-ranks chunks, and a synthesizer writes the answer. A system that retrieves chunks full of pronouns wastes its reasoning budget untangling noise.

It also changes the cost profile — each reasoning step consumes LLM tokens and each re-ranking pass consumes embedding lookups. The economics only work when documents are clean enough that the first pass surfaces the right chunks.

Hybrid Search: Combining Keyword and Semantic Retrieval

Pure vector retrieval struggles with rare proper nouns, product codes, and error strings. Hybrid search pairs vector retrieval with keyword (BM25) search; OpenSearch Service supports it natively, and Bedrock Knowledge Bases can use it via its OpenSearch connector. A fusion algorithm — typically reciprocal rank fusion (RRF) — merges the two ranked lists into one that rewards chunks both semantically relevant and lexically matched.

Hybrid search rewards RAG optimization directly: well-optimized documents have descriptive headings and concrete nouns whose chunks match the exact terms users query for, so they surface twice — vector and keyword sides. Re-ranking adds a precision layer: a cross-encoder (Titan Text Embeddings as a reranker, or Cohere via Bedrock) scores each chunk against the query, and only the top-N (typically five to ten) pass to the generator.

RAG Optimization on AWS: Service Mapping

Amazon Bedrock Knowledge Bases is the managed ingestion-and-retrieval layer: point it at an S3 bucket, choose an embedding model (Amazon Titan Text Embeddings, Cohere Embed, or a partner model), and it handles chunking, embedding, and metadata extraction automatically.

Amazon OpenSearch Service is the vector store most teams pick at scale — k-NN search with HNSW or IVF, native filtered search for tenant-aware retrieval, and CloudWatch monitoring.

Amazon Kendra performs its own semantic indexing with AWS-trained models, excels at enterprise formats (PDF, Word, Confluence, SharePoint), and absorbs document noise more gracefully — so the manual-optimization lift is smaller than a pure vector pipeline, but still the highest-ROI investment in the stack. Kendra also supports extractive answers — snippets pulled from source text that work best when prose is clean and declarative.

Choose Bedrock Knowledge Bases for end-to-end control; Kendra for enterprise search with minimal overhead; OpenSearch Service directly for fine-grained vector index control. Amazon S3 sits underneath all three — a hierarchical bucket layout (e.g. s3://kb-source/products/bedrock/user-guide/) beats a flat prefix, and S3 tags enable tenant-aware filtering.

RAG Optimization in Practice: A Worked Example

BEFORE: “It processes the data and returns it in a format the system can use.” AFTER: “Amazon Bedrock Knowledge Bases processes the raw document chunks and returns structured embeddings that Amazon OpenSearch Service uses for similarity search.” The rewrite replaces “it” with a concrete noun, “data” with “raw document chunks”, and “the system” with the actual service — so the sentence embeds tightly around Bedrock, OpenSearch, embeddings, and similarity search.

The same applies to headings, which drive chunk boundaries. “Step 3 — Configuration” embeds as a generic cluster; “Step 3: Configure the IAM Role for Bedrock Knowledge Bases Ingestion” embeds tightly around IAM, Bedrock, and ingestion — a single rewrite can move a chunk from never-retrieved to top-three. Below is an audit script flagging violations before ingestion.

Before that script can run against production content, the ingestion path itself needs the right permissions. A Bedrock Knowledge Base data source assumes a service role with, at minimum, s3:GetObject and s3:ListBucket scoped to the source prefix, bedrock:InvokeModel for the embedding model, and — when the vector store is Amazon OpenSearch Service — aoss:APIAccessAll or the equivalent fine-grained OpenSearch Serverless data-access policy.

A common failure mode is granting the role bucket-level s3:GetObject but forgetting s3:ListBucket. That combination lets individual objects download fine in testing, because whoever tested it already knew the exact key, but it breaks the bulk-crawl step of a full sync, surfacing only as a partial-ingestion warning days later once the missing documents are noticed by a user, not by a dashboard.

With permissions in place, kick off a sync from the CLI after every batch of document rewrites:

aws bedrock-agent start-ingestion-job \
  --knowledge-base-id KBSTQEXAMPLE \
  --data-source-id DSSTQEXAMPLE \
  --description "Post-audit re-embed after Step 3 heading rewrites"

aws bedrock-agent get-ingestion-job \
  --knowledge-base-id KBSTQEXAMPLE \
  --data-source-id DSSTQEXAMPLE \
  --ingestion-job-id JOBSTQEXAMPLE \
  --query 'ingestionJob.statistics'

The statistics block reports documents scanned, indexed, failed, and deleted for that job. A non-zero numberOfDocumentsFailed almost always traces back to one of two causes: a document exceeding the embedding model’s input token limit — split it before ingestion rather than truncating silently and losing the end of the document — or a transient throttling error against the embedding model.

Bedrock Knowledge Bases retries throttled embedding calls internally, but a job can still surface a failure if retries are exhausted during a burst of concurrent ingestion jobs.

Wire a CloudWatch alarm on numberOfDocumentsFailed greater than zero so a partial sync failure does not silently leave stale chunks serving alongside a handful of freshly re-embedded ones — a confusing state to debug after the fact, since some queries will reflect the rewrite and others will not.

# Audit a corpus for common document-quality issues
import boto3, re
from botocore.config import Config

s3 = boto3.client('s3', config=Config(region_name='us-east-1'))

ANAPHORA_FIRST = re.compile(r'^\s*(It|This|That|They|These|Those)\s', re.I)
VAGUE_HEADING  = re.compile(r'^#+\s*(Step \d+|Section \d+|More info)\s*$', re.I)

def audit_document(text):
    issues = []
    paragraphs = text.split('\n\n')
    for i, p in enumerate(paragraphs):
        words = p.split()
        if len(words) > 110:
            issues.append(f'para {i}: {len(words)} words (max 110)')
        if ANAPHORA_FIRST.match(p):
            issues.append(f'para {i}: starts with pronoun — replace with noun')
    for line in text.splitlines():
        if VAGUE_HEADING.match(line):
            issues.append(f'heading "{line.strip()}" is non-descriptive')
    return issues

# Wire into the S3 PUT event that triggers Bedrock Knowledge Bases ingestion
# Route flagged documents to an SQS review queue before they are embedded

Measuring RAG Optimization Impact

The metrics that matter are recall@K (did the retriever surface the relevant chunk in top-K?), precision@K (how many of top-K are relevant?), and mean reciprocal rank.

Build an evaluation set of two hundred to five hundred questions, run them before and after the rewrite, and have a human or LLM judge label each chunk — a successful pass moves both numbers up by at least ten points.

Bedrock Knowledge Bases exposes retrieval metadata through CloudWatch — use it to find queries that pulled low-similarity chunks and prioritize those documents. Round out the scorecard with latency p95, token cost, and answer faithfulness.

RAG Optimization: Common Mistakes to Avoid

Most RAG accuracy regressions trace back to a handful of repeated, avoidable mistakes rather than a fundamentally hard retrieval problem. RAG optimization work pays off fastest when it targets these first, before reaching for a bigger embedding model or a more elaborate re-ranking pipeline.

• Treating cleanup as a one-time project — audit every document on ingest, not quarterly. Documentation drifts continuously as writers add new pages and edit old ones; a quarterly audit means three months of degraded chunks serving live traffic before anyone notices.
• Optimizing only top documents — the long tail is where users hit dead ends. Teams naturally polish the handful of pages product managers care about and leave hundreds of long-tail runbooks and edge-case docs untouched; those are exactly the documents a support or troubleshooting agent needs most.
• Ignoring images and tables — uncaptioned images and unsummarized tables contribute zero retrieval signal. A screenshot of a console workflow with no alt text is invisible to the embedding model no matter how central it is to the human reader’s understanding.
• Chasing model upgrades over document fixes — a larger embedding masks the problem for a quarter, then it returns. Fix documents first. A bigger embedding model is more forgiving of noisy chunks up to a point, but it does not remove the noise — the underlying documents still need the same rewrite eventually, and by then the corpus has usually grown.
• Skipping the ingestion-job statistics check — a sync that reports success at the console level can still have numberOfDocumentsFailed greater than zero in the job statistics; teams that only watch for a green checkmark miss silent partial failures that leave stale chunks live.
• Changing chunking strategy without re-evaluating — switching from fixed-size to hierarchical or semantic chunking changes chunk boundaries for the entire corpus; deploying the change without comparing recall@K against the old strategy on a held-out evaluation set risks a regression that is only caught in production.

RAG Optimization: Best Practices

• Write for the chunker, not just the reader — every paragraph should remain coherent if extracted alone. Read each paragraph in isolation, with no surrounding context, and ask whether a reader (or a model) landing on it cold would understand what it is about.
• Replace every pronoun with its referent noun, especially in first sentences. First sentences carry the most retrieval weight because they set the topical center of the chunk’s embedding; a pronoun there costs more than one buried mid-paragraph.
• Add a one-sentence summary at the end of each major section. The summary sentence tends to land closest to the section’s topical centroid in vector space, which makes it disproportionately likely to be the sentence that actually gets retrieved for a broad query about that section.
• Run a monthly redundancy audit using k-NN search; consolidate chunks with cosine similarity above 0.95. Treat the audit as a recurring pipeline job, not a manual one-off, so redundancy never has time to compound across a growing corpus.
• Pair every image, table, and diagram with a prose caption or summary. If the image is load-bearing for understanding a procedure, the caption should describe the procedure, not just label the image — “the ingestion pipeline” is weaker than a caption that names the actual services and the order data flows through them.
• Version-control chunking-strategy changes like schema migrations. Record which strategy and parameters produced the currently-live index, keep the evaluation-set scores that justified the last change, and require a new set of recall@K numbers before rolling out the next one.
• Wire the audit script into the same pipeline that triggers ingestion. A CodeBuild or Lambda step that runs the document audit before calling start-ingestion-job catches anaphora, vague headings, and oversized paragraphs before they ever reach the vector store, the same way a linter catches style violations before a build.

RAG Optimization: Frequently Asked Questions

What is the difference between RAG optimization and prompt engineering?

Prompt engineering shapes the instructions and query sent to the LLM at inference time; RAG optimization shapes the source documents before they are ever chunked or embedded. Prompt engineering can compensate for weak retrieval only up to a point — if the top-K chunks the retriever surfaces are noisy or irrelevant, no amount of prompt tuning recovers the missing signal. Fixing document quality upstream is what makes prompt engineering effective downstream.

Does this work apply to Amazon Kendra as well as Bedrock Knowledge Bases?

Yes. Kendra performs its own semantic indexing, but clear headings, self-contained paragraphs, explicit nouns, and section summaries all improve its accuracy. Bedrock Knowledge Bases, OpenSearch Service, Aurora pgvector, and Kendra all benefit — the lift is smaller with Kendra because its indexer normalizes document noise automatically.

How often should I run a RAG optimization audit?

Continuously, not quarterly. Audit every document on ingest — flag pronoun-heavy prose, missing captions, and vague headings before it enters the corpus — plus a monthly redundancy scan. Wire it into your documentation CI/CD pipeline like linting in code.

Can an LLM automate the document rewrite?

Partially. An LLM can do the mechanical work — rewriting anaphora, generating summaries, expanding vague headings — but needs human review for domain accuracy. Safe pattern: LLM-proposed rewrites as pull requests, automated retrieval tests, human approval before merge.

What is the return on investment of document optimization?

The largest single ROI in the retrieval stack — typically fifteen to twenty-five percent accuracy improvements versus single-digit gains from model swaps or prompt tuning. Cost is mostly one-time labor plus a small ongoing audit cost; it pays back inside a quarter.

RAG Optimization: Key Takeaways

• Document quality dominates model quality — a well-optimized corpus with a mid-tier embedding beats a messy corpus with a state-of-the-art model.
• Structure drives chunkability — clear headings, short paragraphs, flat lists, and section summaries produce tight embeddings.
• Anaphora is the largest avoidable noise source — replacing “it”, “this”, and “that” with explicit nouns is the highest-leverage rewrite.
• Non-text content needs text proxies — images, tables, and diagrams must be paired with descriptive captions.
• RAG optimization is continuous — audit on ingest and run a monthly redundancy scan.

RAG optimization is writing documents that survive chunking and embedding with meaning intact. On Bedrock Knowledge Bases, OpenSearch Service, and Kendra, it is the highest-leverage investment in retrieval accuracy. RAG accuracy on AWS Bedrock Knowledge Bases is gated by document quality, not model size: cleaner chunks beat larger context windows.