RAG quality does not start with your vector database or your embedding model — it starts with the document you feed into it. In this hands-on RAG lab you will take a genuinely messy internal policy PDF, restructure it into clean, chunking-friendly sections, load both versions into an Amazon Bedrock Knowledge Base, and run the same test questions against each.
You will see, in real retrieved chunks, exactly why document structure is the highest-leverage lever in any RAG pipeline.
What You’ll Build: A RAG Retrieval Comparison
By the end of this lab you will have two Amazon S3 objects — a poorly-structured policy document and a well-structured rewrite of it — each ingested into its own Bedrock Knowledge Base data source. You will sync ingestion and query both with an identical set of four test questions using the Retrieve and RetrieveAndGenerate APIs. The final artifact: a side-by-side comparison showing why RAG retrieval quality is inseparable from source-document quality.
Prerequisites: an AWS account with Bedrock model access enabled (an embedding model such as Amazon Titan Text Embeddings V2, plus a text model for generation), an S3 bucket, IAM permissions to create a Bedrock Knowledge Base and its OpenSearch Serverless collection, and Python 3.9+ with boto3. No prior Bedrock Knowledge Base experience is required — this lab builds one from scratch.
This lab isolates one variable: both knowledge bases use the identical embedding model, chunking strategy, and maxTokens/overlapPercentage settings. The only thing that differs is the source document text — so if RAG retrieval quality differs, you know it is because of how the document was written, not a hidden config difference. A single rewritten page is enough to see the effect; the whole lab runs in under an hour.
Building the RAG Comparison Lab
The lab has seven parts: write the “before” and “after” documents, upload both to S3, stand up two data sources, configure chunking and sync ingestion, then run identical test questions and compare the retrieved chunks. Every step below includes runnable code. The hypothesis: a document organized into clear, single-topic sections will produce more semantically coherent chunks than continuous prose covering several topics at once.
Step 1 — The “Before”: a realistic messy policy document
Most internal knowledge bases are seeded with documents exactly like this one — a “Remote Work & Expense Policy” written as a single wall-of-text memo, mixing eligibility rules, expense limits, equipment stipends, and approval workflows in the same paragraphs, with the same facts repeated in two different places using slightly different numbers. This is not a contrived worst case; it is what most company wikis and shared drives actually contain.
MessyCo Remote Work and Expense Policy (v3, unstructured)
MessyCo allows employees to work remotely and this policy covers remote work
eligibility as well as expense reimbursement and equipment because these topics
often come up together when someone starts working from home. Employees who
have been with the company for at least 90 days and whose manager approves are
eligible for remote work, and remote employees can expense up to $75 per month
for internet, and if you need a monitor or keyboard the company will reimburse
up to $300 total for home office equipment but you need manager approval first
and receipts must be submitted within 30 days or the expense may be rejected,
also note that new hires in their first 90 days are not eligible for remote
work unless an exception is granted by an SVP, and separately, all expense
reports over $50 require manager approval while reports over $500 require
director approval, and by the way internet reimbursement is actually capped at
$75/month per the finance team's 2024 update though some older documents say
$50/month so check with finance if unsure, and equipment stipends reset every
two years, and travel expenses for occasional in-office visits by remote
employees are reimbursed at the standard mileage rate, and remote employees
must attend in-person meetings quarterly unless their manager waives this,
and finally all expense submissions go through the ExpenseHub tool within 30
days of purchase, receipts required for anything over $25.Notice the structural problems: eligibility rules, expense limits, and approval thresholds are interleaved in one paragraph; the internet stipend is stated twice with conflicting numbers ($75 vs $50); the receipt threshold appears twice ($25 vs $30 days); and no heading signals where one topic ends. A naive chunker splitting this by character count will frequently cut a sentence in half between clauses, producing chunks that answer no single question well.
Step 2 — The “After”: restructured for retrieval
The rewrite keeps every fact from the original but organizes it under clear, topic-scoped headings with one idea per paragraph. Redundant and conflicting statements are resolved to a single source of truth. This is the version a technical writer — or an LLM prompted to “restructure for RAG ingestion, one topic per section, resolve contradictions” — would produce.
MessyCo Remote Work and Expense Policy (v4, restructured)
## Remote Work Eligibility
Employees become eligible for remote work after 90 days of employment,
subject to manager approval. New hires in their first 90 days are not
eligible for remote work unless an SVP grants an exception.
## In-Person Meeting Requirement
Remote employees must attend in-person meetings quarterly. A manager may
waive this requirement on a case-by-case basis.
## Internet Reimbursement
Remote employees may expense up to $75 per month for home internet service.
This is the current, finance-approved rate as of the 2024 policy update and
supersedes any older figure referenced elsewhere.
## Home Office Equipment Stipend
Remote employees may be reimbursed up to $300 total for home office
equipment (monitor, keyboard, chair, etc.), subject to manager approval.
The stipend resets every two years.
## Travel Expense Reimbursement
Occasional in-office travel by remote employees is reimbursed at the
standard IRS mileage rate.
## Expense Submission and Approval Workflow
All expenses are submitted through the ExpenseHub tool. Receipts are
required for any expense over $25 and must be submitted within 30 days of
purchase. Expense reports over $50 require manager approval; reports over
$500 require director approval.Each heading now maps to exactly one retrievable unit of meaning. A fixed-size chunker will still cut mechanically by token count, but because each section is short and self-contained, chunk boundaries are far more likely to fall on section breaks rather than mid-thought — and even when a chunk boundary lands inside a section, the surrounding sentences are about the same topic, so nothing important is orphaned.
Notice also what the rewrite deliberately does NOT add: no new facts, no marketing language, no filler. Restructuring for retrieval is not the same exercise as rewriting for readability or brand voice — it is specifically about giving every fact a clearly labeled, single-topic home so an embedding model can represent it precisely, and giving contradictory statements one resolved answer instead of two.
Step 3 — Upload both versions to Amazon S3
Bedrock Knowledge Bases ingest from a data source, most commonly an S3 bucket. Put each document version in its own prefix so each can be attached to its own data source.
import boto3
s3 = boto3.client("s3", region_name="us-east-1")
BUCKET = "messyco-rag-lab"
s3.put_object(
Bucket=BUCKET,
Key="before/remote-work-policy-v3-messy.txt",
Body=open("remote-work-policy-v3-messy.txt", "rb"),
)
s3.put_object(
Bucket=BUCKET,
Key="after/remote-work-policy-v4-structured.txt",
Body=open("remote-work-policy-v4-structured.txt", "rb"),
)
print("Uploaded both policy versions to s3://%s/{before,after}/" % BUCKET)Keeping the two versions under separate prefixes rather than separate buckets keeps IAM and lifecycle configuration simple — one bucket policy, one set of encryption settings, one place to look — while still giving each Bedrock data source an unambiguous inclusionPrefixes value to scope its crawl to.
Step 4 — Create two Bedrock Knowledge Bases (or one KB, two data sources)
You can either create one knowledge base with two data sources (one pointed at before/, one at after/) or two separate knowledge bases. This lab uses two separate knowledge bases so retrieval results cannot cross-contaminate between the “before” and “after” indexes.
Knowledge base creation itself (vector store, embedding model, IAM role) is a longer one-time setup best done via the console or CDK; the snippet below shows the data-source creation step, which is where chunking strategy is configured, using the bedrock-agent control-plane client. Choose the same embedding model and vector store type (for example, Amazon OpenSearch Serverless) for both KB_BEFORE_ID and KB_AFTER_ID — any difference here becomes a second variable and undermines the comparison.
The IAM service role Bedrock assumes to crawl your data source needs, at minimum, s3:GetObject and s3:ListBucket scoped to the lab bucket, plus permission to invoke the embedding model. If you reuse an existing knowledge base service role from another project, double-check its bucket policy actually covers the new messyco-rag-lab bucket rather than assuming it is unrestricted.
import boto3
bedrock_agent = boto3.client("bedrock-agent", region_name="us-east-1")
def create_data_source(kb_id, name, s3_prefix):
resp = bedrock_agent.create_data_source(
knowledgeBaseId=kb_id,
name=name,
dataSourceConfiguration={
"type": "S3",
"s3Configuration": {
"bucketArn": "arn:aws:s3:::messyco-rag-lab",
"inclusionPrefixes": [s3_prefix],
},
},
vectorIngestionConfiguration={
"chunkingConfiguration": {
"chunkingStrategy": "FIXED_SIZE",
"fixedSizeChunkingConfiguration": {
"maxTokens": 300,
"overlapPercentage": 20,
},
}
},
)
return resp["dataSource"]["dataSourceId"]
before_ds_id = create_data_source("KB_BEFORE_ID", "before-messy-policy", "before/")
after_ds_id = create_data_source("KB_AFTER_ID", "after-structured-policy", "after/")
print("before data source:", before_ds_id)
print("after data source:", after_ds_id)The chunkingConfiguration object accepts a chunkingStrategy of FIXED_SIZE, HIERARCHICAL, SEMANTIC, or NONE — the current, documented options for Bedrock Knowledge Bases data sources. FIXED_SIZE splits content by approximate token count (maxTokens plus overlapPercentage); HIERARCHICAL produces layered parent/child chunks; SEMANTIC groups sentences by similarity using NLP; NONE treats each file as one chunk. This lab uses FIXED_SIZE for both data sources so the ONLY variable compared is document structure.
For reference, HIERARCHICAL requires a fixed two-item levelConfigurations array — one entry per parent/child layer, each with its own maxTokens — plus a shared overlapTokens value:
# Reference only — not used in this lab's comparison, shown for completeness
hierarchical_config = {
"chunkingStrategy": "HIERARCHICAL",
"hierarchicalChunkingConfiguration": {
"levelConfigurations": [
{"maxTokens": 1500}, # parent layer
{"maxTokens": 300}, # child layer
],
"overlapTokens": 60,
},
}
semantic_config = {
"chunkingStrategy": "SEMANTIC",
"semanticChunkingConfiguration": {
"maxTokens": 300,
"bufferSize": 1,
"breakpointPercentileThreshold": 95,
},
}SEMANTIC chunking’s breakpointPercentileThreshold controls how dissimilar consecutive sentences must be before Bedrock inserts a boundary; bufferSize controls how many neighboring sentences are grouped before that comparison happens. This lab holds chunking strategy constant at FIXED_SIZE for both documents on purpose — a well-structured document paired with the simplest chunking strategy already outperforms a messy one, so you do not need semantic chunking to compensate if you fix the source text first.
Step 5 — Sync ingestion for both data sources
Uploading a file to S3 does not automatically index it — you must start an ingestion job so Bedrock crawls, chunks, embeds, and writes vectors to the underlying vector store.
import time
def sync_and_wait(kb_id, ds_id):
job = bedrock_agent.start_ingestion_job(
knowledgeBaseId=kb_id,
dataSourceId=ds_id,
)
job_id = job["ingestionJob"]["ingestionJobId"]
while True:
status = bedrock_agent.get_ingestion_job(
knowledgeBaseId=kb_id,
dataSourceId=ds_id,
ingestionJobId=job_id,
)["ingestionJob"]["status"]
print("ingestion status:", status)
if status in ("COMPLETE", "FAILED"):
return status
time.sleep(10)
sync_and_wait("KB_BEFORE_ID", before_ds_id)
sync_and_wait("KB_AFTER_ID", after_ds_id)The same StartIngestionJob call is what a Sync button click triggers in the console. Incremental syncing means only new, modified, or deleted content is re-crawled after the first sync — useful in production, though for this lab each data source has one file so every sync is a full crawl. During ingestion Bedrock reads each object, chunks it, calls the embedding model once per chunk, and writes the resulting vectors into the vector store.
Watch the get_ingestion_job status transition from STARTING to IN_PROGRESS to COMPLETE; a one-page document should sync in well under a minute. If a job reports FAILED, common causes are an IAM role missing s3:GetObject, a malformed metadata sidecar file, or a model not granted Bedrock access. The GetIngestionJob response includes a failureReasons field naming the specific cause.
Step 6 — Run identical test questions against both knowledge bases
Define one fixed list of test questions and run every question against both knowledge bases using Retrieve, which returns raw chunks and similarity scores without generating an answer — exactly what you want when comparing retrieval quality in isolation. The question set must stay byte-for-byte identical across both knowledge bases; even a small rewording can shift which chunk ranks first purely because of vocabulary overlap, contaminating the comparison.
Choose questions that map to genuinely distinct topics rather than paraphrases of the same fact — this lab’s four questions each target a different section (internet, eligibility, equipment, meeting cadence) so a low score can be traced to one specific chunk boundary rather than a generic embedding problem.
bedrock_runtime = boto3.client("bedrock-agent-runtime", region_name="us-east-1")
TEST_QUESTIONS = [
"How much can a remote employee expense for home internet per month?",
"When is a new hire eligible to start working remotely?",
"What is the reimbursement limit for home office equipment?",
"How often do remote employees need to come into the office?",
]
def retrieve(kb_id, question, top_k=3):
resp = bedrock_runtime.retrieve(
knowledgeBaseId=kb_id,
retrievalQuery={"text": question},
retrievalConfiguration={
"vectorSearchConfiguration": {"numberOfResults": top_k}
},
)
return resp["retrievalResults"]
for q in TEST_QUESTIONS:
print("=" * 70)
print("Q:", q)
print("--- BEFORE (messy doc) ---")
for r in retrieve("KB_BEFORE_ID", q):
print(round(r["score"], 3), "|", r["content"]["text"][:160].replace("\n", " "))
print("--- AFTER (structured doc) ---")
for r in retrieve("KB_AFTER_ID", q):
print(round(r["score"], 3), "|", r["content"]["text"][:160].replace("\n", " "))Run RetrieveAndGenerate as a second pass once you’ve eyeballed the raw chunks, to see the downstream effect on the final answer. A generation model can paper over a mediocre chunk with plausible-sounding phrasing — reading the raw Retrieve output first is what lets you catch that before it reaches an end user.
def retrieve_and_generate(kb_id, question, model_arn):
resp = bedrock_runtime.retrieve_and_generate(
input={"text": question},
retrieveAndGenerateConfiguration={
"type": "KNOWLEDGE_BASE",
"knowledgeBaseConfiguration": {
"knowledgeBaseId": kb_id,
"modelArn": model_arn,
},
},
)
return resp["output"]["text"], resp["citations"]
MODEL_ARN = "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
for q in TEST_QUESTIONS:
answer, citations = retrieve_and_generate("KB_AFTER_ID", q, MODEL_ARN)
print(q, "->", answer)Step 7 — Compare and explain the difference
For the internet-reimbursement question, the messy document’s top chunk is likely to contain both the $75 figure AND the conflicting $50 mention from the same run-on sentence, because they sit only a few words apart — the fixed-size chunker cannot separate them without semantic awareness. The structured document’s “Internet Reimbursement” section chunk contains only the current $75 figure with its supersedes-older-figure clause, because the topic was isolated to one paragraph before chunking ever happened.
For the equipment-stipend question, the messy document’s chunk boundary may cut off the “$300 total” figure entirely if the 300-token window lands between “the company will reimburse” and “up to $300,” pulling in irrelevant eligibility text instead. The structured document’s chunk is short enough that the entire equipment section — heading, dollar amount, and approval requirement — fits inside one chunk with room to spare.
The remote-work eligibility question tells a similar story. In the messy document, “90 days” runs straight into the internet-expense clause, so a chunk boundary can land squarely between the rule and its own exception (“unless an SVP grants an exception”), returning half the rule. In the structured document, the entire “Remote Work Eligibility” section lives in one short paragraph, so a single chunk contains the complete rule.
The quarterly in-person meeting question illustrates redundancy hurting retrieval. In the messy document, the requirement is mentioned once, deep inside a sentence about expense reporting, so its embedding is diluted by unrelated financial vocabulary and may not rank in the top results. In the structured document, “In-Person Meeting Requirement” is its own heading, so its embedding points cleanly at “meeting frequency” with nothing else competing for the chunk’s meaning.
This is the core lesson of the lab: chunk boundaries in RAG systems are mechanical (token counts, sentence splits) unless you use hierarchical or semantic chunking, and even those strategies work best when the underlying document already respects topic boundaries. Restructuring the source document does more to fix bad retrieval than switching chunking strategies or tuning overlapPercentage — because a well-structured document makes ANY chunking strategy work better.
Step 8 — Optional: add metadata and filter at query time
Once you have confirmed the structural difference in raw retrieval, a natural next experiment is adding a metadata sidecar file so you can filter results without creating more knowledge bases. Amazon Bedrock S3 data sources support a separate metadata JSON file per document, using the same key with a .metadata.json suffix, whose fields become filterable at query time.
import json
metadata = {
"metadataAttributes": {
"policy_version": {"value": {"type": "STRING", "stringValue": "v4-structured"}, "includeForEmbedding": False},
"topic": {"value": {"type": "STRING", "stringValue": "remote-work-policy"}, "includeForEmbedding": False},
}
}
s3.put_object(
Bucket=BUCKET,
Key="after/remote-work-policy-v4-structured.txt.metadata.json",
Body=json.dumps(metadata),
)
# Then filter at query time:
resp = bedrock_runtime.retrieve(
knowledgeBaseId="KB_AFTER_ID",
retrievalQuery={"text": "internet reimbursement"},
retrievalConfiguration={
"vectorSearchConfiguration": {
"numberOfResults": 3,
"filter": {"equals": {"key": "policy_version", "value": "v4-structured"}},
}
},
)Re-sync the data source after adding the metadata file so the filterable attributes are indexed, then re-run the test questions with the filter applied. Metadata filtering does not change chunk quality, but it narrows the search space before similarity ranking happens — useful once your knowledge base holds many document versions or departments’ policies side by side.
Common RAG Chunking Mistakes
These are the pitfalls that most commonly sabotage a document-restructuring exercise like this one.
- Setting
maxTokenstoo large (800+) so a single chunk spans multiple unrelated topics — for example mixing the eligibility rule and the expense limit in one embedding — which dilutes the vector and hurts precision for narrow, single-fact questions. - Setting
maxTokenstoo small (under roughly 100) so chunks lose surrounding context, forcing the generation model to guess at meaning from an isolated fragment such as a lone dollar figure with no label explaining what it applies to. - Editing the source document in S3 but forgetting to re-run
start_ingestion_jobafterward — the knowledge base silently keeps serving stale chunks from the previous sync, and query results simply won’t reflect your edit until a new ingestion job completes. - Never tuning
overlapPercentageat all — too little overlap (0%) can split a single fact across two adjacent chunks so that neither chunk alone contains the full statement, while too much overlap (50%+) bloats storage and floods the top-k results with near-duplicate chunks.
Going Further
For the authoritative reference on this topic, see Amazon Bedrock Knowledge Base chunking strategies documentation.
The natural production extension of this lab is to stop eyeballing retrieved chunks and start scoring them. Build a small evaluation harness: for each test question, define the exact ground-truth fact it should surface, then check programmatically whether that fact’s text appears in the top-k retrieved chunks for each knowledge base.
Aggregate a hit-rate percentage across both document versions — this turns “the structured doc looked better” into “the structured doc scored 100% top-3 hit rate versus 50% for the messy doc,” a concrete RAG quality metric you can track as documents change.
GROUND_TRUTH = {
"How much can a remote employee expense for home internet per month?": "$75",
"When is a new hire eligible to start working remotely?": "90 days",
"What is the reimbursement limit for home office equipment?": "$300",
"How often do remote employees need to come into the office?": "quarterly",
}
def hit_rate(kb_id, top_k=3):
hits = 0
for question, expected in GROUND_TRUTH.items():
chunks = retrieve(kb_id, question, top_k=top_k)
text_blob = " ".join(c["content"]["text"] for c in chunks)
if expected.lower() in text_blob.lower():
hits += 1
return hits / len(GROUND_TRUTH)
print("before hit-rate:", hit_rate("KB_BEFORE_ID"))
print("after hit-rate:", hit_rate("KB_AFTER_ID"))A second production concern is automating document-quality checks before ingestion: a lightweight linting step — even a single LLM call per document with a rubric like “does this document use headings, is each paragraph one topic, are there contradictory facts” — flags a document for cleanup before it feeds the knowledge base. Catching a messy document at intake is far cheaper than debugging a confusing RAG answer in production weeks later.
Finally, both embedding calls and vector storage (index size, and OpenSearch Serverless OCU-hours) scale with total chunk count. Well-structured documents with less redundant text produce fewer, denser, more information-rich chunks for the same source material — a real cost saving at scale, not merely a quality improvement.
Frequently Asked Questions
Why not just increase chunk overlap instead of restructuring the document?
Overlap helps preserve context across a chunk boundary, but it cannot fix contradictory facts stated in the same sentence, and it increases the number of chunks stored and searched, which adds cost without addressing the root cause.
Which chunking strategy should I default to for a new knowledge base?
FIXED_SIZE is a solid, predictable default for most text documents. Reach for HIERARCHICAL when documents have a strong nested structure (long manuals with sections and subsections) and SEMANTIC when documents mix many topics loosely and you want NLP-driven boundaries rather than fixed token counts.
Do I need to delete and recreate the data source every time the source document changes?
No. Re-upload the changed object to the same S3 key and call StartIngestionJob again. Amazon Bedrock’s incremental syncing detects the change and re-crawls only what changed since the last sync.
Is Retrieve or RetrieveAndGenerate the right API for this kind of comparison?
Start with Retrieve — it returns the raw ranked chunks and similarity scores with no generation model in the loop, isolating retrieval quality. Use RetrieveAndGenerate afterward to see how retrieval differences actually change the final answer text a user would see.
Does document restructuring replace the need for good chunking configuration?
No — they are complementary. A badly chunked well-structured document will still underperform a well-chunked well-structured document. Restructuring raises the ceiling on how good retrieval CAN be; chunking configuration determines how close to that ceiling you actually get.
RAG retrieval quality is won or lost before a single embedding is ever computed. By rewriting one messy policy document into clearly headed, single-topic sections and loading both versions into separate Bedrock Knowledge Base data sources with identical FIXED_SIZE chunking, you saw directly in the retrieved chunks — not just in theory — why document structure is the highest-leverage input to any RAG system, ahead of chunking strategy tuning or model choice.