Business Challenge
This opens the AI and ML block, and it starts where the failures actually are rather than where the attention is.
A retrieval-augmented system answers a question confidently and wrongly. The fix is looked for in the prompt, then in the model, then in the temperature — the three things a developer can see and change quickly.
None of them is usually the cause. The passage containing the answer was not retrieved, so the model was asked to produce something it was never shown. That failure has a specific and unhelpful signature: it looks exactly like a model that is not good enough.
“Default chunking: Splits content into text chunks of approximately 300 tokens. The chunking process honors sentence boundaries.”
Three hundred tokens is roughly a paragraph. If the answer to a question is established across two paragraphs — a condition stated in one and its exception in the next — then no single chunk contains it. Each chunk holds half, so each scores moderately, and the retriever returns neighbours of the answer rather than the answer.
The model then does what it is asked: it produces a fluent response from adjacent material. There is no error anywhere in the pipeline.
FixSize chunks against the shape of your answers, not against a default.
The obvious response — make chunks bigger — degrades the other half. AWS puts the whole trade in one sentence: “Small text embeddings are more precise, but retrieval aims for comprehensive context.”
A large chunk embeds the average of everything in it, so its vector drifts away from any specific question. A small chunk matches sharply and arrives without the surrounding material that makes it mean anything.
This is not a tuning problem with a right answer. It is a genuine tension, which is why hierarchical chunking exists: retrieve on the small chunk, return the large one.
FixMatch on children, answer from parents. It is the only option that does not force the trade.
Chunking happens at ingestion. By the time anyone is debugging a bad answer, the chunk boundaries are historical — they are not in the request, not in the response, and not in the prompt.
Every other knob is live and adjustable: prompt, model, temperature, number of results. The one that decides what is findable is fixed, and changing it means re-ingesting everything.
That asymmetry is why teams spend weeks on prompts. The prompt is what is in front of them.
FixBefore touching the prompt, retrieve without generating and read what came back.
Architecture
The pipeline is short and the failure is almost always in the first half of it.
Ingestion is three steps, and AWS describes them precisely: Bedrock “splits your documents or content into manageable chunks”, the chunks “are then converted to embeddings and written to a vector index… while maintaining a mapping to the original document”, and “the vector embeddings allow the texts to be quantitatively compared.”
That mapping back to the source is what makes citations possible — knowledge bases can “include citations in the generated response so the original data source can be referenced and accuracy can be checked” — and it is the single most useful feature for diagnosing exactly the failure above. A citation tells you which chunk was retrieved, which tells you whether the answer was ever in the context.
The four strategies, and what each one trades
Default is ~300 tokens on sentence boundaries. Sensible, and a paragraph.
Fixed-size lets you set tokens per chunk and “an overlap percentage between consecutive chunks”. Overlap is the cheap mitigation for the split-answer problem: a boundary that cuts an answer in half still leaves one chunk containing both halves, if the overlap is wide enough.
No chunking treats “each document… as a single text chunk”,
and carries a consequence worth knowing before choosing it: “you cannot view page number in
citation or filter by the x-amz-bedrock-kb-document-page-number metadata
field.” The citation survives; the precision of it does not.
Semantic splits on meaning rather than length, using a buffer size where “a buffer size of 1 results in 3 sentences (current, previous and next sentence) to be combined and embedded”, and a breakpoint percentile threshold where “a higher threshold results in fewer chunks and typically larger average chunk size.” It is the only strategy that costs money to run: “There are additional costs to using semantic chunking due to its use of a foundation model.”
“During retrieval, the system initially retrieves child chunks, but replaces them with broader parent chunks so as to provide the model with more comprehensive context.” Match precisely, answer broadly. The consequence is easy to misread as a bug: “Since child chunks get replaced by parent chunks during retrieval, the returned number of results might be less than the requested amount.” Ask for five results, receive three, because two children shared a parent. That is correct behaviour, and it will look like retrieval failing.
Why This Architecture Holds Up
The vector store is the decision people over-think
Customer-managed knowledge bases support “Amazon OpenSearch Serverless, Amazon Aurora, and Amazon Neptune”, and choosing between them absorbs a lot of design time.
It rarely changes answer quality. All three store vectors and return nearest neighbours; what differs is operational shape and what else you can do with the data alongside — a graph in Neptune, relational joins in Aurora, existing search infrastructure in OpenSearch. A bad chunking strategy produces bad answers on all three equally.
The genuine architectural fork is earlier: managed or customer-managed. With Managed, “Amazon Bedrock manages the underlying data ingestion, indexing, storage, and retrieval infrastructure”. With customer-managed, “you set up and manage your own RAG pipeline, including the vector store… and have full control.”
Choosing customer-managed for control costs capability, not just effort: “several capabilities such as third-party connectors, document-level permissions and native AgentCore Gateway integration are only available for Managed Knowledge Bases.” Document-level permissions is the one to notice. A RAG system over internal content usually needs retrieval to respect who is asking, and that is available on one side of the fork only. Deciding "we'll manage it ourselves for flexibility" can quietly decide that access control is now yours to build.
Some constraints only appear in combination
Individually sensible choices can conflict, and AWS documents one pairing explicitly: “Hierarchical chunking is not recommended when using S3 vector bucket as your vector store,” because “when using high number of tokens for chunking (over 8000 tokens combined), you may run into metadata size limitations.”
Hierarchical is the best answer to the precision-versus-context tension, and S3 vector buckets are the cheapest place to keep vectors. Choosing both is reasonable in isolation and unsupported together.
There is a similar shape in multimodal. “Text chunking strategies apply only to text documents” — for audio, video and images, “chunking occurs at the embedding model level.” With Nova embeddings, duration is configurable “from 1-30 seconds (default: 5 seconds)”, and “for video files, only the video chunk duration applies, even if the video contains audio.” A carefully tuned text strategy does nothing to a media corpus.
Key Architecture Decisions
| Decision | Choice | Reasoning |
|---|---|---|
| Where to debug first | Retrieval, before the prompt | A wrong answer from missing context is indistinguishable from a weak model. |
| Chunk size | From the shape of your answers | The 300-token default is a paragraph. An answer spanning two lives in no chunk. |
| Precision versus context | Hierarchical chunking | Retrieves on children, returns parents — the only strategy that does not force the trade. |
| Cheap mitigation | Overlap percentage | A wide enough overlap means a split answer still appears whole in one chunk. |
| Citations | Always on | They are the only view of which chunk was retrieved, which is the whole diagnosis. |
| Managed vs customer-managed | Managed unless a named requirement | Document-level permissions and connectors exist on one side of the fork only. |
| Vector store | Pick on operations, not accuracy | All three return nearest neighbours. Chunking decides the answers. |
| Semantic chunking | When boundaries matter more than cost | It invokes a foundation model at ingestion, so it is the only strategy with a running bill. |
The one measurement worth building first
Before evaluating models, evaluate retrieval alone: take a set of real questions, record which chunks come back, and check by hand whether the answer is present in them. That number — how often the answer was retrievable at all — is the ceiling on everything downstream.
No prompt raises it. No model raises it. It is decided by chunking, and it can be measured without generating a single response, which makes it both the most useful metric and the easiest one to collect.
Closing Thought
RAG is named for what it does last. The retrieval is in the name, and the attention still goes to the generation, because generation is what produces something visible to argue with.
The consequence is a characteristic pattern: weeks of prompt iteration against a corpus that was chopped into paragraphs on day one by a default nobody revisited, and a conclusion that the model is not good enough. Sometimes the model is not good enough. Far more often it was handed the two paragraphs either side of the answer.
AWS's own sentence is the most useful thing in the documentation and the easiest to read past: small text embeddings are more precise, but retrieval aims for comprehensive context. Those two goals are in direct conflict, the resolution is a chunking decision, and it is made once, before anybody has asked the system a question.
AI & ML — Bedrock model choice, provisioned throughput and cost: why on-demand and provisioned are different products rather than two prices, what a model unit commits you to, and the quota that decides whether a launch is possible in your Region at all.
Comments