Bedrock Knowledge Base Learns to Choose Search Depth: From 0 LLM Calls to Agentic RAG

In the morning, a bank employee asks an internal AI assistant which documents are needed to open a sole proprietor account. The answer sits in a single checklist. In the afternoon, the same employee asks how the requirements differ for sole proprietors, LLCs and partnerships with several owners. That answer is spread across multiple policy documents. It is one assistant, but the questions differ widely in difficulty and cost. If you run a heavy agentic loop for both, you overpay and lose time to latency. If you run a single search for both, you lose accuracy on the hard question.

AWS is closing this gap in Amazon Bedrock Managed Knowledge Base. Previously, only agentic mode was available through AgenticRetrieveStream. Now there are two more.

One parameter instead of two APIs

Everything is controlled through maxAgentIteration. The range is now 0–5, and the value determines how the request behaves.

0 — retrieval only. The service runs one guaranteed search and generates nothing. There are no foundation model calls, so it is essentially the equivalent of the regular Retrieve API. This mode suits teams that have already moved to AgenticRetrieveStream and don't want to switch to another API just for a query without generation. If you set generateResponse=true with a value of 0, you get a 4xx error: generating an answer requires an LLM.

1 — classic RAG. One search, one model call to produce the answer, no planning and no query rewriting. The article's example: "Which documents are needed to open a sole proprietor business account?" The checklist is in the knowledge base, and a planner would only add latency. If most of your traffic consists of direct, clearly scoped questions, AWS hints that this should be your default.

2–5 — agentic mode. The model breaks the question into subqueries, searches for each one, assesses whether what it found is sufficient, and if necessary refines the search or pulls in a whole document. Values 2–5 work as before, so existing requests won't break.

How iterations are counted

The rule is simple: one iteration equals one foundation model call. The final iteration always goes to generating the answer. So a value of n gives up to n−1 planning rounds, and each round is a query rewrite plus a search.

There is a nuance. Speculative search, which runs before the first planning step, does not count as an iteration. It does not consume the limit and can only speed up the request.

Separately, AWS stresses that a single search does not mean a single knowledge base. All three modes, including 0 and 1, work with multiple knowledge bases. Agentic mode adds question decomposition, evaluation of the evidence found and iterative search, and multi-knowledge-base support does not depend on it.

In the bank example, it looks like this. For the sole proprietor checklist question, set maxAgentIteration=1 and generateResponse=true. For a comparison of sole proprietors and LLCs with non-resident checks, set 4, which means up to three planning rounds. When the employee needs the text of the policy itself rather than a summary, set 0 and generateResponse=false, and the application shows passages with links to sources. Switching from an agentic call to plain RAG is a one-number edit in the JSON.

Limitations and what to do in practice

First, and most important: the choice of mode stays on your application's side. There is no automatic routing by question complexity. You need either a simple router (by scenario type, by heuristic, or with a small classifier) or a fixed mode assigned to a specific screen or feature.

Second: the response format in mode 0 differs from direct Retrieve. Deduplicated passages arrive in the stream's result event without score and location fields. If your interface relies on a relevance score or the exact location of a fragment, check this before migrating.

Third: the bank examples in the article are illustrative. AWS provides no measurements of latency, cost or quality. The claim that single-shot reduces latency and model spend compared with agentic mode is logical, but there are no numbers behind it. Don't take it on faith.

A practical plan:

• Build a set of real questions with reference answers drawn from your documents.
• Run it through mode 1 and through your current agentic configuration on the same knowledge base.
• Measure latency and cost per request, and check that the answers are correct and supported by the retrieved passages.
• Be sure to include questions that require assembling an answer from several documents. That is where single-shot retrieval usually falls short.

If agentic mode is currently your default, chances are that on simple queries you are paying for planning that adds nothing. Start with a low value for simple traffic and raise it where accuracy on hard questions matters more than speed. The iteration limit also needs tuning on your own data: more iterations don't always mean a better answer.