Model Routing for Coding Agents: A Practical Guide

A coding agent is the most demanding LLM consumer most teams will ever run. A single user request, fix this failing test, becomes a loop of model calls: plan the approach, call a tool, read its output, call another, and eventually write the answer, with a set of auxiliary calls around it, summarizing long command output, writing the commit message, titling the session, compacting history. Fifteen model calls per visible task is unremarkable. The calls differ wildly in difficulty, and an agent wired to one frontier model pays frontier prices for every one of them, including the call that shortens a stack trace.

This guide covers the practical shape of model routing for coding agents, using Inferbase as the working example. The boundary is the same one our RAG guide draws: your agent framework, its tools, its sandbox, and its orchestration stay yours, and only the model calls cross it. What is specific to agents is the split this guide is built around: the tool-calling loop pins a concrete model, the auxiliary calls auto-route, and the reasons for that split matter more than the recipe. If tool calling itself is new territory, what is tool calling covers the mechanics this article builds on.

Anatomy of an agent's traffic

Before wiring anything, inventory what your agent actually sends. The traffic falls into two lanes with different requirements.

The tool-calling loop is the calls that carry tools: the model reads the conversation, decides whether to call a tool or answer, and emits either content or structured tool calls. These calls are behaviorally sensitive. The prompt, the tool schemas, and the model's tool-calling habits are tuned together, and swapping the model mid-loop changes how often it calls tools, how well it formats arguments, and how it recovers from tool errors. They are also the calls where a weak model fails silently: a malformed argument string does not look like a failure until the tool rejects it three iterations later.

The auxiliary calls carry no tools and are mostly mechanical: compress this diff, summarize this test output, write a one-line commit message, generate a title. Chat frontends already make two or three of these background calls per visible turn; agents make more. They are high-volume, low-difficulty, and nobody has tuned a prompt against one model's habits for them. Paying frontier prices here is pure waste, and this is exactly the heterogeneous traffic routing monetizes.

The integration rule that falls out of this inventory is short enough to remember: pin the loop, route the rest.

Why the loop pins: an honest constraint

On this API, that rule is enforced rather than merely advised. A request that carries tools with model set to auto is refused with a structured error:

{
  "detail": "Tool calls require a pinned model today; smart routing with tools is coming. Pin any served model (see GET /models) to use tools."
}

The refusal is a design position. Routing a tool call responsibly requires per-model evidence about tool-calling reliability, and that evidence is harder to come by than generation quality scores: a model can benchmark well on code and still emit garbage arguments under a five-field nested schema. Serving that model silently would fail in the worst way, plausible-looking calls that are wrong. Until the router carries a real tool-call quality signal, tool traffic pins, and the error says so instead of degrading quietly. Auto-routing with tools is planned, beginning with capability-filtered custom pools where the caller ranks the candidates.

For the auxiliary lane the constraint does not exist, because those calls carry no tools, and everything in the migration guide about auto, optimization modes, and model pools applies to them unchanged.

Wire the tool loop

The endpoint is OpenAI-compatible, so the loop is the standard shape: send tools, execute what the model calls, append results as tool messages, repeat until finish_reason is no longer tool_calls.

import json
from openai import OpenAI

client = OpenAI(
    api_key="inf_your_api_key",
    base_url="https://api.inferbase.ai/api/v1/inference",
)

TOOLS = [{
    "type": "function",
    "function": {
        "name": "run_tests",
        "description": "Run the test suite for a path and return the failures.",
        "parameters": {
            "type": "object",
            "properties": {
                "path": {"type": "string", "description": "Test file or directory"},
            },
            "required": ["path"],
        },
    },
}]

AGENT_MODEL = "Qwen/Qwen3-235B-A22B-Instruct-2507-tput"

messages = [
    {"role": "system", "content": "You are a coding agent. Verify your conclusions with the tools."},
    {"role": "user", "content": "tests/test_parser.py started failing on main. Find out why."},
]

while True:
    response = client.chat.completions.create(
        model=AGENT_MODEL,
        messages=messages,
        tools=TOOLS,
        temperature=0.0,
    )
    choice = response.choices[0]
    messages.append(choice.message.model_dump(exclude_none=True))

    if choice.finish_reason != "tool_calls":
        break

    for call in choice.message.tool_calls:
        args = json.loads(call.function.arguments)
        result = run_in_sandbox(call.function.name, args)  # your execution layer
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": result,
        })

print(choice.message.content)

Three details in that loop repay attention. The assistant message carrying tool_calls legitimately has no content, and it must be appended to the history as-is, because the model's next turn needs to see its own calls. Every tool message requires the tool_call_id it answers and string content; the pairing is how the model matches results to calls when it made several. And tool_choice is available when you need to force behavior, "required" to guarantee a call, "none" to suppress tools for one turn, or a specific function object to demand one tool by name, all forwarded to the provider with OpenAI function-calling semantics.

Requests accept up to 128 tool definitions, 200 messages, and 128K max_tokens. Long-running agents hit the message cap before any other; that is the point at which your framework's history compaction should engage, which is itself an auxiliary call that belongs on auto.

Choose the pin deliberately

Which model should anchor the loop? The model catalog records function-calling capability per model, which gives you the shortlist: served models that declare the capability, in the size tier your tasks need.

What the catalog cannot tell you is how a model behaves under your schemas, and tool-calling reliability genuinely varies between models that look equivalent on paper. The selection method that works is unglamorous: take two or three candidates, run your actual agent loop against each with temperature at zero, and count malformed arguments, hallucinated tool names, and premature answers. An afternoon of that beats any benchmark table.

Two disciplines from elsewhere on this blog apply with extra force. Pin the most specific model identifier available, because an agent loop is exactly the tuned-prompt situation where a silently swapped model does the most damage. And record why the pin was chosen next to the agent's prompt, since the two were tuned together and will be revisited together.

A pinned call through the routing layer still buys provider failover: if the pinned model is served by more than one provider, an outage fails over behind the same request rather than in your retry logic. Pinning constrains which model serves, not which provider survives.

Route the auxiliary calls

Everything without tools goes to auto, and the volume makes it worth doing properly rather than incidentally. Give the auxiliary lane its own API key with its own default optimization mode, and let the router classify each call:

aux = OpenAI(
    api_key="inf_aux_lane_key",
    base_url="https://api.inferbase.ai/api/v1/inference",
)

summary = aux.chat.completions.create(
    model="auto",
    messages=[{
        "role": "user",
        "content": f"Summarize this test output in three lines, keep file paths exact:\n\n{raw_output}",
    }],
    extra_body={"routing": {"optimize": "cost"}},
)

The per-call split in a typical agent:

One structural caveat: this split is only available if your framework routes its sub-calls through configurable clients. Most serious agent frameworks expose exactly that, a model setting for the main loop and a separate cheap-model setting for auxiliary work, and the pattern above is what those two settings should point at. A framework that attaches the full tool set to every call, including the mechanical ones, forces everything onto the pin; the fix is in the framework configuration, not the API.

If compliance constrains where code may be sent, set a model_pool on the auxiliary key so auto only ever picks approved models, and note that agent prompts carry source code by construction: the account-level zero-retention setting, which disables even the opt-in content stores, is the right posture before an agent touches a private repository.

Handle streaming tool calls

Agent frontends stream, and tool calls stream as fragments: delta.tool_calls entries keyed by index, the function name arriving in an early fragment and the argument string accumulating across the rest, terminated by finish_reason: "tool_calls". The accumulation pattern:

stream = client.chat.completions.create(
    model=AGENT_MODEL,
    messages=messages,
    tools=TOOLS,
    stream=True,
)

calls: dict[int, dict] = {}
for chunk in stream:
    if not getattr(chunk, "choices", None):
        continue  # routing/usage metadata events, not content deltas
    delta = chunk.choices[0].delta
    if delta.content:
        render(delta.content)
    for frag in delta.tool_calls or []:
        slot = calls.setdefault(frag.index, {"id": "", "name": "", "arguments": ""})
        if frag.id:
            slot["id"] = frag.id
        if frag.function and frag.function.name:
            slot["name"] = frag.function.name
        if frag.function and frag.function.arguments:
            slot["arguments"] += frag.function.arguments

for slot in calls.values():
    args = json.loads(slot["arguments"])  # parse only after the stream ends

Parse the arguments JSON only after the stream completes; fragments are not individually valid JSON. The choices guard matters on this endpoint for the reason covered in the migration guide: a routed stream opens with a routing metadata event and closes with a usage event, and both surface through the SDK as chunks without choices.

Latency compounds in loops, so measure it honestly

Agent calls are sequential by nature: the model cannot read a tool result before the tool runs. Whatever per-call overhead your stack adds is therefore multiplied by loop length, and a routing layer earns a hard look on precisely this axis.

The routing decision here is a single classifier call, not a separate LLM round trip, and because every response's usage event carries latency_ms and ttft_ms, its cost on your own traffic is a logging query rather than a claim to trust. Generation dominates an agent's wall clock, which cuts the other way too: the real latency lever in an agent is serving the mechanical calls on small fast models instead of a frontier model, and that is the auxiliary lane's latency axis doing its work.

The claim is verifiable rather than promotional: every streamed response's usage event carries latency_ms and ttft_ms per call, so a before-and-after comparison of your agent's step timings is a logging query, not a benchmark project.

Audit the loop like you audit the code

An agent that edits code needs a paper trail, and the routed responses carry one. Non-streaming responses disclose the served model in the model field; streams open with the routing event carrying the served model, the task and complexity classification, and scored alternatives for auto-routed calls; every call yields a usage event with tokens and cost_micros. Log these keyed to your agent's own step identifiers and two questions become queries: what did this task cost per step, and did any auxiliary call land on a model outside the approved pool. With a key per lane, the dashboard's per-key breakdown answers the surface-level version of the same questions without any client work.

What to respect, stated plainly

Boundaries worth knowing before the first run rather than after. Tool-carrying requests must pin; auto returns the structured 400 quoted above until routed tool support ships. The router makes no claims about a pinned model's tool-calling quality, the catalog records the capability and your own loop testing establishes reliability.

Structured output is a separate mechanism from tools: response_format with json_object or a forwarded json_schema covers extract-this-shape cases without tool machinery, and structured outputs covers choosing between them. Requests cap at 200 messages, 128 tools, and 128K max_tokens. And nothing on the serve path stores prompt or response content; the zero-retention account flag additionally disables the two opt-in stores, which is the setting to confirm before an agent reads private code.

Rolling it out

1. Create two keys: one for the agent loop, one for auxiliary calls, so spend and behavior attribute per lane from the first request.
2. Shortlist pinned candidates from the catalog's function-calling data and run your loop against each at temperature zero; pick on malformed-argument rate, not reputation.
3. Point the loop at the pin and verify a full multi-tool task end to end, including the assistant tool_calls message round-tripping through your history handling.
4. Move the auxiliary calls to auto one call site at a time, watching the served-model spread and per-call cost in the usage events.
5. Set the model pool and zero-retention posture before the agent touches a private repository.

The API reference documents the full tool-calling surface, and the model catalog is where pin candidates get shortlisted. The integration is a configuration change; the discipline is in choosing the pin with evidence and letting the router prove its work per call.