Model Routing for Agentic RAG: A Practical Guide

Classic retrieval-augmented generation (RAG) settled into a fixed pipeline, retrieve once, generate once, and its weakness is well documented: the pipeline retrieves whatever the raw user question surfaces, whether or not that is what the answer needs. A question like "how did our refund policy change between the 2024 and 2026 versions" needs two targeted retrievals and a comparison, and a single similarity search over the raw question serves neither half well. The field's response has been to put the model in charge of retrieval itself, an approach that research lines like ReAct and Self-RAG formalized and most serious RAG deployments now borrow from.

That shift has a cost structure worth taking seriously. Agentic RAG replaces one model call per query with three to eight: plan the retrieval, assess the evidence, possibly loop, then synthesize. The calls differ sharply in difficulty, and a deployment that sends all of them to one frontier model multiplies its inference bill by the loop length. This guide covers where model routing fits in an agentic RAG system, using Inferbase as the working example. It builds directly on our classic RAG guide, which covers the boundary, the pool, and the privacy posture; this article covers what changes when the model drives retrieval, and the two implementation shapes that determine how much of the loop can auto-route.

What changes when the model drives retrieval

In classic RAG, the model appears once, at the end. The routing decision is singular: which model generates. Agentic RAG inserts the model before retrieval and between rounds, and each insertion is a distinct call with its own difficulty profile.

The planner reads the user question and emits retrieval instructions, typically two to four sub-queries, sometimes a query rewrite or a filter. This is structurally simple work: no synthesis, no long context, a short JSON output. The assessment step reads what retrieval returned and answers a narrow question, is this enough to answer, or what is missing, which is similarly short and similarly simple.

The synthesis reads everything the rounds accumulated, possibly conflicting passages from different document versions, and writes the answer. This is the step that resembles classic RAG generation on a hard query, and it is where capability genuinely matters.

The spread between those steps is the routing opportunity. A deployment we would call typical runs two planner-class calls and one assessment per question before synthesis; on one frontier model, three quarters of the calls are paying for capability the step cannot use. Routing prices each step at what it needs, which is the same argument as routing generation by query difficulty, applied per call instead of per query.

The two shapes, and why the choice decides your routing

Agentic RAG is implemented in two recognizable shapes, and which one you pick determines how much of the loop can auto-route.

The staged pipeline keeps orchestration in your code. Each step is an ordinary completion: the planner is a call that returns JSON, retrieval is your code against your index, assessment is another short call, synthesis is the final one. No step carries tool definitions, so every step can use auto with its own optimization axis.

The tool-driven loop hands the model a search tool and lets it decide when and what to retrieve, the shape a coding agent uses for its sandbox. It is more flexible on open-ended questions, the model can react to what it finds mid-flight, but on this API tool-carrying requests must pin a concrete model, for the reasons the coding-agent guide covers in depth: routing a tool call honestly requires tool-call quality evidence the router does not claim to have yet, so auto with tools returns a structured 400 rather than degrading silently.

The practical recommendation: start staged. Most agentic RAG workloads are decompose-retrieve-assess-synthesize with a bounded round count, which the staged shape expresses directly, routes fully, and debugs step by step. Reach for the tool-driven loop when questions genuinely require the model to choose retrieval strategy dynamically, and accept the pin as the cost of that flexibility.

Wire the staged shape

The planner is the step teams most often overpay for, and the fix is one call with a JSON response format on the cost axis:

import json
from openai import OpenAI

client = OpenAI(
    api_key="inf_your_api_key",
    base_url="https://api.inferbase.ai/api/v1/inference",
)

def plan_retrieval(question: str) -> list[str]:
    response = client.chat.completions.create(
        model="auto",
        messages=[{
            "role": "user",
            "content": (
                "Decompose this question into 2-4 search queries for a document index. "
                'Return JSON: {"queries": [...]}\n\n' + question
            ),
        }],
        response_format={"type": "json_object"},
        extra_body={"routing": {"optimize": "cost"}},
        temperature=0.0,
    )
    return json.loads(response.choices[0].message.content)["queries"]

response_format with json_object guarantees the output parses; it does not guarantee the shape, so validate the key and retry once on a miss, the distinction structured outputs covers. A parse failure on a planner call is cheap and recoverable, which is exactly why this step tolerates auto-routing where a tool call would not.

Assessment follows the same pattern, a short call on the latency axis that decides whether to loop:

def evidence_sufficient(question: str, passages: list[str]) -> dict:
    response = client.chat.completions.create(
        model="auto",
        messages=[{
            "role": "user",
            "content": (
                "Question: " + question + "\n\nRetrieved passages:\n"
                + "\n---\n".join(passages)
                + '\n\nCan the question be answered from these passages alone? '
                'Return JSON: {"sufficient": true/false, "missing": "..."}'
            ),
        }],
        response_format={"type": "json_object"},
        extra_body={"routing": {"optimize": "latency"}},
        temperature=0.0,
    )
    return json.loads(response.choices[0].message.content)

Synthesis is where the axis flips to quality, and where the constraints from the classic RAG guide apply unchanged: min_context_window sized to your accumulated evidence, the model pool as the compliance boundary, and a prompt that instructs answering from the provided passages:

def synthesize(question: str, evidence: list[str]) -> str:
    response = client.chat.completions.create(
        model="auto",
        messages=[
            {"role": "system", "content": "Answer strictly from the provided passages. Cite passage numbers."},
            {"role": "user", "content": format_evidence(evidence) + "\n\nQuestion: " + question},
        ],
        extra_body={"routing": {"optimize": "quality", "min_context_window": 32000}},
    )
    return response.choices[0].message.content

The orchestration between these calls, run the sub-queries against your index, cap the rounds, carry the evidence forward, is ordinary code and stays yours. Two rounds is a sensible default cap; the assessment step exiting early is the common case, and an uncapped loop is how a confused planner turns one question into a bill.

Wire the tool-driven shape

When the workload justifies it, the loop is the coding-agent pattern with retrieval as the tool:

SEARCH_TOOL = [{
    "type": "function",
    "function": {
        "name": "search_documents",
        "description": "Search the document index. Returns the top passages.",
        "parameters": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
    },
}]

LOOP_MODEL = "Qwen/Qwen3-235B-A22B-Instruct-2507-tput"  # pinned: tools require it

The loop mechanics, appending the assistant's tool_calls message, answering each call with a role: "tool" message carrying its tool_call_id, looping until finish_reason is no longer tool_calls, are identical to the coding-agent loop and covered there, including streaming fragments and how to choose the pin with your own testing rather than benchmark reputation. What remains routable in this shape is everything outside the loop: a pre-loop query rewrite, a post-loop answer condensation for display, follow-up question generation. The split is the same rule at a different boundary: pin what carries tools, route what does not.

Embeddings on the same API

Agentic RAG's retrieval half needs embeddings twice: once to build the index, then per sub-query at run time. Both can run through the same OpenAI-compatible client against the /embeddings endpoint:

index_vectors = client.embeddings.create(
    model="Qwen/Qwen3-Embedding-4B",
    input=chunk_batch,  # up to 512 inputs per request
)
vectors = [item.embedding for item in index_vectors.data]  # order matches input order

Embeddings are pinned-only by design, there is no auto, and the reason is an invariant worth internalizing rather than working around: vector similarity is only defined within one embedding space, so the query-time model must be byte-for-byte the same as the index-time model. A router that swapped embedding models per request would corrupt every index it touched. Record the embedding model id as index metadata, and treat a model change as a reindexing project. For background on what these vectors are, what are embeddings covers the fundamentals; the served embedding models and their prices are in the model catalog.

The economics of the loop

A worked example makes the routing case concrete. Take a knowledge-base assistant doing 10,000 questions a month, averaging two planner calls, one assessment, and one synthesis per question, with the per-call token counts that pattern produces: planner and assessment calls run a few hundred tokens each, synthesis runs a few thousand with the accumulated evidence in context.

On a single frontier-class model at typical frontier pricing, all four calls pay the same rate, and the three cheap calls together often cost as much as the synthesis, they are shorter, but there are three of them. Routed, the planner and assessment calls land on small fast models at one to two orders of magnitude lower price per token, and the synthesis keeps the strong model. The arithmetic lands in the range of 40 to 70 percent off the total, dominated by how heavy your synthesis context is relative to the loop calls. Your own numbers are computable rather than estimable: every response carries a usage event with cost_micros per call, so a week of logs answers the question for your actual traffic.

The same logging settles latency. Rounds are sequential, so the loop's wall clock is the sum of its calls, and the per-call latency_ms and ttft_ms in the usage events show which stage dominates. In practice the fix is rarely about the routing layer's own overhead, a single lightweight classification step whose cost shows up in those same per-call fields, and usually about letting the cheap steps run on fast small models, which the latency axis does per call.

The privacy boundary moves with the loop

One consequence of agentic RAG deserves explicit attention: retrieved content now crosses the model boundary multiple times per question, in the assessment calls and again in synthesis, not just once at the end. The posture from the classic RAG guide applies to every one of those calls: a curated model_pool on the key restricts all of them to compliance-approved models, and the account-level zero-retention setting keeps prompt and response content unstored on every call in the loop. Nothing new needs configuring, the same key-level controls cover the extra calls, but a privacy review that approved "one generation call sees retrieved documents" should be updated to reflect the loop.

Verify the loop is routing sanely

Each routed call disclosed its decision: the model field and X-Inferbase-Model header on plain responses, the routing event with task, complexity, and scored alternatives on streams. In an agentic system, log these keyed to the stage name, and two checks matter in the first week. Planner and assessment calls should land on small models nearly always; if they draw strong models, the classifier is reading your prompt as harder than it is, and trimming boilerplate from the instruction usually fixes it. Synthesis should show a spread that tracks question difficulty; a synthesis lane that never draws a strong model on quality means the pool's ceiling is too low for your hardest questions.

Rolling it out

1. Start from a working classic RAG deployment per the RAG guide; agentic RAG is an upgrade to it, not a replacement for it.
2. Add the planner as one staged call with response_format on the cost axis, and measure answer quality on your evaluation set before adding rounds; decomposition alone captures much of the benefit.
3. Add assessment and cap rounds at two. Log stage-keyed routing and usage events from the first request.
4. Embed through the pinned embeddings endpoint and record the model id in index metadata.
5. Move to the tool-driven shape only if staged decomposition demonstrably fails your question mix, and pick the pin by testing per the coding-agent guide.

The API reference documents the routing parameters, response formats, and embeddings endpoint. The pattern to keep: agentic RAG multiplies model calls, and multiplied calls are exactly the traffic shape where per-call routing stops being an optimization and becomes the difference between a viable bill and an inflated one.