Home Blog AWS Architecture Series #61 — The model never saw the answer…
AWS Architecture AWS Architecture Series

AWS Architecture Series #61 — The model never saw the answer

A RAG system returns a confident, wrong answer. The investigation goes to the prompt and then to the model, because those are the parts that are visible. The cause is usually that the passage containing the answer was never retrieved — and no prompt and no model can recover from text they were not given.

Verified against current vendor documentation on 23 September 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

Business Challenge

This opens the AI and ML block, and it starts where the failures actually are rather than where the attention is.

A retrieval-augmented system answers a question confidently and wrongly. The fix is looked for in the prompt, then in the model, then in the temperature — the three things a developer can see and change quickly.

None of them is usually the cause. The passage containing the answer was not retrieved, so the model was asked to produce something it was never shown. That failure has a specific and unhelpful signature: it looks exactly like a model that is not good enough.

1The answer does not fit in a chunk

“Default chunking: Splits content into text chunks of approximately 300 tokens. The chunking process honors sentence boundaries.”

Three hundred tokens is roughly a paragraph. If the answer to a question is established across two paragraphs — a condition stated in one and its exception in the next — then no single chunk contains it. Each chunk holds half, so each scores moderately, and the retriever returns neighbours of the answer rather than the answer.

The model then does what it is asked: it produces a fluent response from adjacent material. There is no error anywhere in the pipeline.

Fix

Size chunks against the shape of your answers, not against a default.

2Precision and context pull in opposite directions

The obvious response — make chunks bigger — degrades the other half. AWS puts the whole trade in one sentence: “Small text embeddings are more precise, but retrieval aims for comprehensive context.”

A large chunk embeds the average of everything in it, so its vector drifts away from any specific question. A small chunk matches sharply and arrives without the surrounding material that makes it mean anything.

This is not a tuning problem with a right answer. It is a genuine tension, which is why hierarchical chunking exists: retrieve on the small chunk, return the large one.

Fix

Match on children, answer from parents. It is the only option that does not force the trade.

3The decision is made once and then hidden

Chunking happens at ingestion. By the time anyone is debugging a bad answer, the chunk boundaries are historical — they are not in the request, not in the response, and not in the prompt.

Every other knob is live and adjustable: prompt, model, temperature, number of results. The one that decides what is findable is fixed, and changing it means re-ingesting everything.

That asymmetry is why teams spend weeks on prompts. The prompt is what is in front of them.

Fix

Before touching the prompt, retrieve without generating and read what came back.

Architecture

The pipeline is short and the failure is almost always in the first half of it.

Diagram: where a retrieval-augmented generation pipeline actually fails on AWS. Ingestion runs left to right: a document is split into chunks, each chunk is converted to an embedding and written to a vector index while a mapping back to the original document is maintained. At query time the question is embedded and compared against that index, the top matches are placed into the prompt, and the model generates an answer. A panel marks the first half of the pipeline as the part that decides what can ever be found, and notes that by the time anyone debugs a wrong answer the chunk boundaries are historical and appear in no request or response. The chunking strategies are compared: default chunking splits content into approximately 300 tokens while honouring sentence boundaries, fixed-size chunking lets you set tokens per chunk and an overlap percentage, no chunking treats each document as a single chunk and removes page numbers from citations, semantic chunking uses a buffer size where a value of one embeds three sentences together and a breakpoint percentile threshold where a higher value produces fewer and larger chunks at additional foundation model cost, and hierarchical chunking retrieves small child chunks but returns their larger parents. A panel records AWS's own statement of the central tension, that small text embeddings are more precise but retrieval aims for comprehensive context, and notes that hierarchical chunking is the only strategy that does not force a choice between the two, at the cost that the number of returned results may be fewer than requested because children are replaced by parents.
Everything before the arrow decides what can ever be found. Everything after it is what teams spend their time on.

Ingestion is three steps, and AWS describes them precisely: Bedrock “splits your documents or content into manageable chunks”, the chunks “are then converted to embeddings and written to a vector index… while maintaining a mapping to the original document”, and “the vector embeddings allow the texts to be quantitatively compared.”

That mapping back to the source is what makes citations possible — knowledge bases can “include citations in the generated response so the original data source can be referenced and accuracy can be checked” — and it is the single most useful feature for diagnosing exactly the failure above. A citation tells you which chunk was retrieved, which tells you whether the answer was ever in the context.

The four strategies, and what each one trades

Default is ~300 tokens on sentence boundaries. Sensible, and a paragraph.

Fixed-size lets you set tokens per chunk and “an overlap percentage between consecutive chunks”. Overlap is the cheap mitigation for the split-answer problem: a boundary that cuts an answer in half still leaves one chunk containing both halves, if the overlap is wide enough.

No chunking treats “each document… as a single text chunk”, and carries a consequence worth knowing before choosing it: “you cannot view page number in citation or filter by the x-amz-bedrock-kb-document-page-number metadata field.” The citation survives; the precision of it does not.

Semantic splits on meaning rather than length, using a buffer size where “a buffer size of 1 results in 3 sentences (current, previous and next sentence) to be combined and embedded”, and a breakpoint percentile threshold where “a higher threshold results in fewer chunks and typically larger average chunk size.” It is the only strategy that costs money to run: “There are additional costs to using semantic chunking due to its use of a foundation model.”

Hierarchical is the one that refuses the trade — and it has a tell

“During retrieval, the system initially retrieves child chunks, but replaces them with broader parent chunks so as to provide the model with more comprehensive context.” Match precisely, answer broadly. The consequence is easy to misread as a bug: “Since child chunks get replaced by parent chunks during retrieval, the returned number of results might be less than the requested amount.” Ask for five results, receive three, because two children shared a parent. That is correct behaviour, and it will look like retrieval failing.

Why This Architecture Holds Up

The vector store is the decision people over-think

Customer-managed knowledge bases support “Amazon OpenSearch Serverless, Amazon Aurora, and Amazon Neptune”, and choosing between them absorbs a lot of design time.

It rarely changes answer quality. All three store vectors and return nearest neighbours; what differs is operational shape and what else you can do with the data alongside — a graph in Neptune, relational joins in Aurora, existing search infrastructure in OpenSearch. A bad chunking strategy produces bad answers on all three equally.

The genuine architectural fork is earlier: managed or customer-managed. With Managed, “Amazon Bedrock manages the underlying data ingestion, indexing, storage, and retrieval infrastructure”. With customer-managed, “you set up and manage your own RAG pipeline, including the vector store… and have full control.”

And that fork is not symmetrical

Choosing customer-managed for control costs capability, not just effort: “several capabilities such as third-party connectors, document-level permissions and native AgentCore Gateway integration are only available for Managed Knowledge Bases.” Document-level permissions is the one to notice. A RAG system over internal content usually needs retrieval to respect who is asking, and that is available on one side of the fork only. Deciding "we'll manage it ourselves for flexibility" can quietly decide that access control is now yours to build.

Some constraints only appear in combination

Individually sensible choices can conflict, and AWS documents one pairing explicitly: “Hierarchical chunking is not recommended when using S3 vector bucket as your vector store,” because “when using high number of tokens for chunking (over 8000 tokens combined), you may run into metadata size limitations.”

Hierarchical is the best answer to the precision-versus-context tension, and S3 vector buckets are the cheapest place to keep vectors. Choosing both is reasonable in isolation and unsupported together.

There is a similar shape in multimodal. “Text chunking strategies apply only to text documents” — for audio, video and images, “chunking occurs at the embedding model level.” With Nova embeddings, duration is configurable “from 1-30 seconds (default: 5 seconds)”, and “for video files, only the video chunk duration applies, even if the video contains audio.” A carefully tuned text strategy does nothing to a media corpus.

Key Architecture Decisions

Decision Choice Reasoning
Where to debug first Retrieval, before the prompt A wrong answer from missing context is indistinguishable from a weak model.
Chunk size From the shape of your answers The 300-token default is a paragraph. An answer spanning two lives in no chunk.
Precision versus context Hierarchical chunking Retrieves on children, returns parents — the only strategy that does not force the trade.
Cheap mitigation Overlap percentage A wide enough overlap means a split answer still appears whole in one chunk.
Citations Always on They are the only view of which chunk was retrieved, which is the whole diagnosis.
Managed vs customer-managed Managed unless a named requirement Document-level permissions and connectors exist on one side of the fork only.
Vector store Pick on operations, not accuracy All three return nearest neighbours. Chunking decides the answers.
Semantic chunking When boundaries matter more than cost It invokes a foundation model at ingestion, so it is the only strategy with a running bill.

The one measurement worth building first

Before evaluating models, evaluate retrieval alone: take a set of real questions, record which chunks come back, and check by hand whether the answer is present in them. That number — how often the answer was retrievable at all — is the ceiling on everything downstream.

No prompt raises it. No model raises it. It is decided by chunking, and it can be measured without generating a single response, which makes it both the most useful metric and the easiest one to collect.

Closing Thought

RAG is named for what it does last. The retrieval is in the name, and the attention still goes to the generation, because generation is what produces something visible to argue with.

The consequence is a characteristic pattern: weeks of prompt iteration against a corpus that was chopped into paragraphs on day one by a default nobody revisited, and a conclusion that the model is not good enough. Sometimes the model is not good enough. Far more often it was handed the two paragraphs either side of the answer.

AWS's own sentence is the most useful thing in the documentation and the easiest to read past: small text embeddings are more precise, but retrieval aims for comprehensive context. Those two goals are in direct conflict, the resolution is a chunking decision, and it is made once, before anybody has asked the system a question.

Next in this series

AI & ML — Bedrock model choice, provisioned throughput and cost: why on-demand and provisioned are different products rather than two prices, what a model unit commits you to, and the quota that decides whether a launch is possible in your Region at all.

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent