Coding agents

An agent loop where the planning call and the tool-calling steps are routed on what each one is, the loop stays on one model while a tool call is open, and the tools the agent may call are governed by policy. Built from scratch with a sandbox, four tools and one session per run.

What you build

A service that takes a task against a repository, runs an agent loop in a sandbox, and returns a diff with its reasoning. The loop makes two kinds of call: a plan and a final answer that ask for quality, and tool steps that ask for latency. Every call sends the model as auto under one routing policy whose pool holds only models with confirmed tool support, carries the same session identifier so the run is one record, and is bounded by the policy's caps on requests and spend per session. The tools the agent may call are an allow-list in that policy, taken from a dry run to enforced.

When you are done

• A service that turns a task and a repository into a diff, with each step's model, cost and tool calls on record
• Planning and synthesis on the strongest model in the pool, tool steps on the fastest, without naming either in code
• A tool surface that is a list in the policy: dry-run first, then enforced
• Runs that end on your rule or on the policy caps, never on a forgotten counter
• An evaluation that reports pass rate, cost and steps per task, for auto against a pinned model

The components

Task intake

An HTTP endpoint or a queue consumer that receives a task, a repository and a base commit, and starts a run with a fresh session identifier.

Sandbox

A container per run with the repository checked out at the base commit, where tools read files, run tests and apply patches. Nothing the model asks for runs outside it.

Tools

Four functions declared on every tool step: list_files, read_file, run_tests, apply_patch. Each is a thin wrapper over the sandbox with a size cap on what it returns.

The loop

Plan, then tool steps until the model answers without a tool call or the step limit is reached, then the final answer. Each call carries the routing object for its kind.

Inferbase

The key, the routing policy with its rule pool, task rule, allow-list and session caps, and the chat completions endpoint.

Run record

Each call's request id, model, cost and tool calls, stored with the run; the same facts appear under Usage as one agent run grouped by the session.

Evaluation harness

A set of tasks with hidden tests. Runs each through the service and scores whether the tests pass, how many steps it took, and what it cost.

The problem

A coding agent makes many calls per task, and they are not the same kind of call.

One task is a plan call, several tool-choice calls, a few summaries of tool output, and a final answer. The plan and the final answer need the strongest reasoning in the pool. Choosing the next tool and formatting its arguments does not, and a model that is good at it answers in a fraction of the time. A loop that sends every step to one frontier model pays the plan price twenty times and waits for it twenty times, because sequential calls add up.

The opposite mistake is as common: a cheap model everywhere, until the plan is wrong and the loop spends its budget executing it. And whichever model is chosen, a tool call that comes back mangled costs a retry at best and a wrong edit at worst.

The third problem is the tool surface. The list of functions an agent may call lives in the application code, changes with every deploy, and is seen by nobody who is responsible for what the agent may do to a repository.

This guide routes each call on what it is. The plan asks for quality; the tool steps ask for latency; every call sends the model as auto under a policy that only admits models whose tool support is confirmed. A tool loop stays on the model that opened it, so a tool result never lands on a model that did not ask for it. The tools the agent may call are an allow-list in the policy, and a run that does not stop is stopped by the policy step and spend caps.

Who has it

• A team running an agent loop on one frontier model and watching the bill and the wall clock grow with it
• An internal coding assistant whose tool surface should be a list someone approved
• Anyone who needs each run on record: which model took which step, and what it cost

Architecture

System overview

Every component, grouped by who runs it. The product sits at the generation call; everything in your lanes is yours and unchanged by this design.

One request, end to end

Orchestration and tool execution stay in your stack. Inferbase chooses the model per call, keeps a tool loop on one model, and governs which tools the model may call.

Deployment

Where each part runs, and what talks to what over the network.

What stays in your stack

• The loop and its stopping rule
• Tool implementations and the sandbox they run in
• Repository access and credentials
• The prompts for plan and final answer

Technology stack

One choice per layer, the alternative that was considered, and the reason. Swap any row; the Inferbase call does not change.

What carries it

The Inferbase features this design rests on. Each exists in the product today and is linked to its documentation.

Per-request objective and task

The routing object on each call narrows the policy for that call: optimize for quality on the plan, for latency on tool steps, with the task fixed to coding so the router does not have to guess. Documentation

Tool-capable models only

With tools on the request and the model set to auto, routing is limited to models whose tool support is confirmed and whose tool-use quality is measured. A model with unknown tool support is excluded. Documentation

The loop stays on its model

A conversation mid tool loop stays on the model that made the call, and falls back only within that model, so a tool result is read by the model that asked for it. Documentation

Tool governance

The policy allow-lists the tool names a key may send, from Any tool through Dry run to Enforce, and records every tool a project sends and when it was first seen. Documentation

Session caps

One session identifier per run groups its calls under one agent run in Usage and subjects it to the policy caps on requests and spend per session. A loop that does not terminate is terminated. Documentation

Task rules

A rule for coding tries a named model first for that task alone, leaving every other task on the objective; a rule's model still has to clear the evidence bar. Documentation

Pinned-model fallback

A step pinned to one model falls back the way the policy says: the same model on another provider, a model you choose, auto, or none. Documentation

The Responses API and the Anthropic shape

An agent built on the OpenAI Agents SDK or the Anthropic SDK sends the shape it already sends. Tool definitions, tool choice, parallel calls and tool results work in either. Documentation

Build it step by step

Eight steps from an empty service to a governed loop. The code is Python; the two-call loop is also given in TypeScript in the section after the policy, and that version was run against the API. Each step ends with the thing it turns on.

Before you start

• An Inferbase account, a project, and a key with a spend cap for this service.
• A container runtime the service can drive (Docker or Podman), and a credential that can clone the repositories the agent will work on, scoped to read and to push branches.
• Python 3.12, the openai package, and a test runner the repositories use.

import subprocess

CAP = 8_000


class Sandbox:
    def __init__(self, container: str):
        self.container = container  # started by intake: the repo checked out at the base commit

    def _run(self, *argv: str) -> str:
        out = subprocess.run(["docker", "exec", self.container, *argv], capture_output=True, text=True, timeout=120)
        text = (out.stdout + out.stderr).strip()
        return text if len(text) <= CAP else text[-CAP:] + "\n[... truncated]"

    def list_files(self, pattern: str = "**/*") -> str:
        return self._run("sh", "-c", f"cd /repo && git ls-files | grep -E '{pattern}' | head -200")

    def read_file(self, path: str) -> str:
        return self._run("sh", "-c", f"cd /repo && cat -- '{path}'")

    def run_tests(self, selector: str = "") -> str:
        return self._run("sh", "-c", f"cd /repo && python -m pytest -q {selector} 2>&1 | tail -80")

    def apply_patch(self, diff: str) -> str:
        return self._run("sh", "-c", f"cd /repo && printf '%s' '{diff}' | git apply --3way - && git diff --stat")


TOOLS = [
    {"type": "function", "function": {"name": "list_files", "description": "List files matching a pattern.", "parameters": {"type": "object", "properties": {"pattern": {"type": "string"}}, "required": []}}},
    {"type": "function", "function": {"name": "read_file", "description": "Read a file from the repository.", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}},
    {"type": "function", "function": {"name": "run_tests", "description": "Run the test suite, or a selector.", "parameters": {"type": "object", "properties": {"selector": {"type": "string"}}, "required": []}}},
    {"type": "function", "function": {"name": "apply_patch", "description": "Apply a unified diff to the repository.", "parameters": {"type": "object", "properties": {"diff": {"type": "string"}}, "required": ["diff"]}}},
]

from openai import OpenAI

client = OpenAI(api_key="inf_your_api_key", base_url="https://api.inferbase.ai/api/v1/inference")


def plan(task: str, run_id: str) -> tuple[str, dict]:
    completion = client.chat.completions.create(
        model="auto",
        max_tokens=400,
        messages=[
            {"role": "system", "content": "You plan the work of a coding agent. Name the files to read, the likely cause, and how to verify. Five lines at most."},
            {"role": "user", "content": task},
        ],
        extra_body={"routing": {"optimize": "quality", "task": "coding", "session": run_id}},
    )
    return completion.choices[0].message.content or "", {"request_id": completion.id, "model": completion.model_extra["routing"]["model"]}

import json
from openai import APIStatusError


def run_loop(sandbox: Sandbox, task: str, plan_text: str, run_id: str, record: list[dict], max_steps: int = 20) -> str:
    messages = [
        {"role": "system", "content": "You are a coding agent. Read before you change. Run the tests after a patch. When the tests pass, answer with the diff and two sentences on the cause."},
        {"role": "user", "content": f"{task}\n\nPlan:\n{plan_text}"},
    ]
    for _ in range(max_steps):
        try:
            step = client.chat.completions.create(
                model="auto",
                max_tokens=800,
                messages=messages,
                tools=TOOLS,
                extra_body={"routing": {"optimize": "latency", "task": "coding", "session": run_id}, "extended_thinking": False},
            )
        except APIStatusError as exc:
            if exc.status_code in (402, 429):  # the session cap, or the key's limit: the run is over
                return f"stopped by the gateway: {exc.message}"
            raise  # a 403 naming a tool outside the allow-list is a bug in the tool list, not a retry
        choice = step.choices[0]
        record.append({"request_id": step.id, "model": step.model_extra["routing"]["model"], "tool_calls": [c.function.name for c in choice.message.tool_calls or []]})
        messages.append(choice.message)
        if choice.finish_reason != "tool_calls":
            return choice.message.content or ""
        for call in choice.message.tool_calls:
            args = json.loads(call.function.arguments or "{}")
            result = getattr(sandbox, call.function.name)(**args)
            messages.append({"role": "tool", "tool_call_id": call.id, "content": shorten(result, run_id)})
    return "step limit reached"

def shorten(text: str, run_id: str, limit: int = 4_000) -> str:
    if len(text) <= limit:
        return text
    summary = client.chat.completions.create(
        model="auto",
        max_tokens=300,
        messages=[
            {"role": "system", "content": "Summarise tool output for a coding agent: which tests failed, the error messages, the file and line numbers. Keep every identifier."},
            {"role": "user", "content": text[-12_000:]},
        ],
        extra_body={"routing": {"optimize": "cost", "task": "summarisation", "session": run_id}, "extended_thinking": False},
    )
    return "[summarised]\n" + (summary.choices[0].message.content or "")

def explain(diff: str, tests: str, run_id: str) -> str:
    completion = client.chat.completions.create(
        model="auto",
        max_tokens=1_500,
        messages=[
            {"role": "system", "content": "Write the pull request description: the cause, the change, how it was verified. Plain sentences."},
            {"role": "user", "content": f"Diff:\n{diff}\n\nTests:\n{tests}"},
        ],
        extra_body={"routing": {"optimize": "quality", "task": "coding", "session": run_id}},
    )
    return completion.choices[0].message.content or ""

# The patch step, pinned: the same model every time, the same model on another provider if it fails.
step = client.chat.completions.create(
    model="your-pinned-model-id",  # a catalog id; the policy's pinned-model fallback decides what happens if it fails
    max_tokens=800,
    messages=messages,
    tools=[TOOLS[3]],  # apply_patch only
    extra_body={"routing": {"session": run_id}},
)

import uuid


def run_task(task: str, repo: str, base: str) -> dict:
    run_id = f"run-{uuid.uuid4().hex[:8]}"  # one session: one record, one set of caps
    sandbox = start_sandbox(repo, base)
    record: list[dict] = []
    try:
        plan_text, plan_meta = plan(task, run_id)
        record.append({**plan_meta, "kind": "plan"})
        outcome = run_loop(sandbox, task, plan_text, run_id, record)
        diff = sandbox._run("sh", "-c", "cd /repo && git diff")
        tests = sandbox.run_tests()
        explanation = explain(diff, tests, run_id) if diff else outcome
        return {"run_id": run_id, "diff": diff, "explanation": explanation, "calls": record}
    finally:
        stop_sandbox(sandbox)

The routing policy

A rule pool of tool-capable models, Balanced as the default with the plan asking for quality per request, an enforced tool allow-list once Dry run has shown the list is complete, and caps per session.

Create it under Policies in the portal and attach it to the key the application uses; the steps are in Create a routing policy.

The call

The two kinds of call in one loop. The plan asks for quality; the tool steps ask for latency and carry the tools; both carry the same session identifier so the run is one record. Tool execution is a stub here, your sandbox in production; the full build above adds the sandbox, the summaries and the record.

import json
from openai import OpenAI

client = OpenAI(api_key="inf_your_api_key", base_url="https://api.inferbase.ai/api/v1/inference")

RUN = "run-7f3a"  # one session id per run: one record, one set of caps
TOOLS = [{
    "type": "function",
    "function": {
        "name": "read_file",
        "description": "Read a file from the repository.",
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"],
        },
    },
}]


def read_file(path: str) -> str:  # your sandbox; a stub here
    return "def parse_dates(rows):\n    first = rows[0]  # IndexError on []\n    ..."


task = "The test test_parse_dates fails on an empty input. Find the cause in src/dates.py and propose the fix."

# 1. The plan: the strongest model the pool holds for coding.
plan = client.chat.completions.create(
    model="auto",
    max_tokens=300,
    messages=[{"role": "user", "content": f"Plan the steps to resolve this, in three lines:\n{task}"}],
    extra_body={"routing": {"optimize": "quality", "task": "coding", "session": RUN}},
)
print("plan by", plan.model_extra["routing"]["model"])

# 2. The tool steps: fast, tool-capable, and the loop stays on the model that opened it.
messages = [
    {"role": "system", "content": "You are a coding agent. Use tools to read code before answering."},
    {"role": "user", "content": f"{task}\n\nPlan:\n{plan.choices[0].message.content}"},
]
for _ in range(6):
    step = client.chat.completions.create(
        model="auto",
        max_tokens=400,
        messages=messages,
        tools=TOOLS,
        extra_body={"routing": {"optimize": "latency", "task": "coding", "session": RUN}},
    )
    choice = step.choices[0]
    messages.append(choice.message)
    if choice.finish_reason != "tool_calls":
        print("answer by", step.model_extra["routing"]["model"])
        print(choice.message.content)
        break
    for call in choice.message.tool_calls:
        args = json.loads(call.function.arguments)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": read_file(**args)})

Both snippets were run against api.inferbase.ai on 2026-10-13: the plan routed to Kimi K3 or DeepSeek V3.2, the tool steps called read_file and answered on the model that opened the loop, and the whole run took about a minute non-streamed.

Open in PlaygroundAPI reference

Security and privacy

An agent that edits code is an agent that runs code. The controls are about where it runs, what it may call, and what it may spend.

Execution is confined

Tools run in a container per run with the repository and nothing else: no secrets mounted, network limited to the repository host. A model that asks for a shell gets no tool by that name.

The tool list is policy

The allow-list in the routing policy, enforced, refuses any name outside it before a model is chosen. The list has a revision history and applies to every key under the policy; a developer cannot widen it from the loop.

Secrets stay out of the prompt

Tool results are text from the sandbox. Environment variables, tokens and keys are never readable by a tool, so they are never in a prompt and never in a request log.

Repository access is scoped

The clone credential reads the repository and pushes branches. It cannot merge, delete or reach other repositories; a human merges the pull request.

Spend is bounded twice

The policy caps requests and spend per session, so one run cannot loop forever; the key caps spend per day, so many runs cannot either.

Every call is recorded

Each call's request id reads back its decision; the tool calls per run are under Usage; a tool seen for the first time raises an event. What the agent did is never a mystery after the fact.

Evaluation

A task set with hidden tests is the oracle. Run every task through the service and score three things per task: whether the hidden tests pass, how many steps it took, and what it cost from the receipts. Then run the same set with the plan and the steps pinned to one model and compare.

Pass rate

The fraction of tasks whose hidden tests pass on the diff the agent produced. The only quality number that matters for a coding agent.

Steps and wall time

From the run record: calls per run and elapsed time. A loop that reads the right file first is a loop that ends sooner; the plan quality shows up here.

Cost per task

Sum of cost_micros over the run's calls. Compare it by call kind: the plan and the explanation should dominate; if tool steps do, the latency objective is not landing on cheap models and the pool needs a look.

Auto against pinned

The same set with every call pinned to your current model. The difference in pass rate is the quality cost or gain of routing; the difference in cost per task is the saving.

Read the decisions

Group the auto run's calls by model. The plan should land on one or two strong models, the steps on one or two fast ones; anything else is a pool or rule to adjust.

results = []
for case in tasks:  # {"task": ..., "repo": ..., "base": ..., "hidden_tests": "tests/test_hidden.py"}
    run = run_task(case["task"], case["repo"], case["base"])
    passed = hidden_tests_pass(case["repo"], case["base"], run["diff"], case["hidden_tests"])
    cost = sum(call.get("cost_usd", 0) for call in run["calls"])
    results.append({"run_id": run["run_id"], "passed": passed, "steps": len(run["calls"]), "cost": cost})

print(f"pass rate {sum(r['passed'] for r in results) / len(results):.0%}, "
      f"median steps {sorted(r['steps'] for r in results)[len(results) // 2]}, "
      f"mean cost {sum(r['cost'] for r in results) / len(results):.4f} USD")

Operations

A run is the unit of everything: debugging, cost, governance. Keep the run record and read the gateway's view of the same run beside it.

Agent runs under Usage

Each session id is one run: every call with its model, task, cost, and the tools it declared and called. The run record you keep should agree with it; when they differ, the gateway is right.

Tools first seen

When a project sends a tool name for the first time, an event records it. A new verb in an agent is visible the day it appears, which is when to decide whether it belongs on the list.

Dry-run reports

A tool served outside a dry-run list is recorded with the tool and the key. When the reports go quiet for a week, enforce.

Which model takes which call

Group calls by routing objective and model. If tool steps are landing on a slow deployment, the latency evidence for that route is the thing to look at; a task rule is the fast fix.

Stopped runs

Runs that ended on a 429 or 402 are runs the caps caught. Read them: a cap that fires often is either too low or catching a loop that needs a better stopping rule.

Policy changes

The pool, the rule and the allow-list have a revision history. Change them with the evaluation in hand and rerun it.

Scaling and limits

Runs are independent, so the service scales by running more of them. The limits are the key, the sandbox host and the history length.

Real time, measured

Not an interactive-latency product, and it should not be sold as one. The verified two-call loop took about a minute non-streamed on 2026-10-13; a real task is several minutes of tool steps. What matters is throughput and the per-step overhead: the decision is tens of milliseconds per call, and each tool step waits on the provider's first token, 1.5 to 3.5 s under the latency objective on 2026-10-14. Run tasks in the background and report when done.

Concurrent runs

A queue in front of the intake and a sandbox per run. Each run is a few dozen requests over a few minutes; a key at 600 requests a minute carries many runs at once.

History length

Tool output capped at 8,000 characters and summarised above 4,000 keeps a 20-step run inside a 128k window. For longer runs, summarise the history itself at a checkpoint with a cost-objective call and carry the summary forward.

Prompt caching

The system prompt and the early history repeat on every step. Turn prompt caching on in the policy; the decision says whether the chosen provider caches.

Sandbox cost

A container per run is the price of safety. Pool warm containers for the repositories that run most, and destroy a container at the end of its run without exception.

Parallel tool calls

A model may return several tool calls at once. Execute them concurrently in the sandbox when they are reads, serially when one is a patch.

Rate limits

Raise the key's limit before the queue does it for you; a 429 from the key ends a run, as the loop treats it.

Troubleshooting

What goes wrong when this is built, what it means, and what to do. Each one was met while building or running it.

Extending

The loop is small and the seams are clear. Each extension changes one component.

Repository tools through MCP

Declare your Git platform's MCP server under Connections and let the policy allow tools by server; or let Inferbase call it, so the credential leaves the service. The tools and MCP guide covers both.

The OpenAI Agents SDK

Point the SDK at the Responses endpoint with the base URL changed; it sends the shape the endpoint takes, and the session rides in a header.

Human approval gates

Pause before apply_patch and ask a developer. The loop is yours; a gate is an await on a decision, with the session kept so the run resumes on its model.

A review agent

The same loop with read-only tools and a prompt that reviews a diff instead of writing one. Same policy, a different allow-list.

Many repositories

The sandbox is per run; the service is per team. A pool per team, if their compliance differs, is a policy per key.

Your own keys

Build the pool from a provider key or cloud account. The loop does not change; the policy decides whose credential serves each call.

What was measured

What one step costs before its first token, and what a short run took, on the public API. A loop pays the first-token wait on every step, so it is the figure that matters here.

Measured from our own infrastructure in India against api.inferbase.ai on 2026-10-14, three runs each, client-side wall time including the network hop (about 0.25 s each way), thinking off. Every response returns its own figures.

Further reading

• The Inferbase quickstarts repository; the coding-agent folder for this guide is planned
• Guide: model routing for coding agents
• Tools and MCP on the API
• Tool governance
• The routing object on a request
• Routing policies: rules, fallback, session
• Client compatibility: the Agents SDK and Anthropic shapes
• Build guide: agents with tools and MCP

Frequently asked questions