Voice agent

Speech in, speech out, on one key: transcription, a routed answer kept short and fast, and speech synthesis streamed so the first audio arrives before the sentence is finished. Built as a turn service with a browser client, for turn-based use: an IVR, an in-app helper, a voicemail that answers. Measured at 7 to 13 seconds a turn today, it is not a real-time phone agent, and the guide says what would make it one.

What you build

A service that takes a caller's audio for one turn, transcribes it, answers in one or two spoken sentences from a known domain, and streams the answer back as audio while it is still being synthesised. A browser client records a turn, sends it, and plays PCM as it arrives; a telephony bridge can do the same. The three calls go to Inferbase on one key: transcription and speech take the first model of their type in your policy pool, the answer is routed on latency with thinking off, and every call returns its model and its cost. Optional tools let the answer act on your systems; the session keeps a call on one model across turns.

When you are done

• A turn service: audio in, transcript, answer, audio out, with every stage timed, so you know what a turn costs before you build the client
• A browser client that records, sends, and plays the answer as it streams
• Answers that are short, fast and from your domain, with the model named on every turn
• A conversation that stays on one model across turns, bounded by a per-call spend cap
• Three receipts per turn: seconds transcribed, tokens answered, characters spoken, each with its cost

The components

Browser client

Records a turn with the MediaRecorder API, posts it, and plays the streamed PCM through the Web Audio API as chunks arrive.

Turn service

A FastAPI endpoint per turn: transcribe, answer, speak. Streams the answer text and then the audio bytes to the client.

Conversation state

The transcript so far, kept by the service per call id, sent on each answer call. The call id is the routing session.

Domain prompt and tools

The system prompt that keeps answers short and spoken, and optional tools (order lookup, booking) the answer may call.

Inferbase

The key, the routing policy with an ordered pool (one transcription model and one speech model at the top of their types, fast chat models beside them), and the three endpoints.

Receipts

Per turn: the transcription's seconds, the answer's tokens and model, the speech's characters, each with cost and latency, stored with the call.

Evaluation

A set of recorded utterances with expected transcripts and answers; scored on word error, answer correctness, and time to first audio.

The problem

A voice turn is three calls, and the caller hears every millisecond of all of them.

A spoken question has to be transcribed, answered and spoken back, and the person on the line is waiting through all three. A text chat can take four seconds to start; a voice agent that takes four seconds to start answering has lost the caller. The three calls also come from three vendors in most stacks, with three keys, three bills and three places where a request can fail.

The answer itself has a different shape from a text answer. It is read aloud, so it must be short, and it must come from a model that starts fast. A reasoning model that thinks for seven hundred tokens before saying hello is the wrong model for a phone call, however good its answer.

Speech synthesis is the slowest of the three when it is done whole: a sentence is synthesised in full before the first byte returns. The streaming form returns bytes as they are made, and which formats actually stream early depends on the provider.

This guide makes the three calls on one key. Transcription and speech take the first model of their type in your policy pool, in the order you set, because nothing measures audio models against each other and a policy choice is better than a guess. The answer is routed on latency with thinking off and a tight output cap. Speech is requested in the streaming form so the first audio arrives while the rest is still being synthesised. Measured end to end, a turn takes 7 to 13 seconds today, which makes this a turn-based assistant and not a real-time conversation; the scaling section says exactly where the time goes and what would close the gap.

Who has it

• A phone or in-app voice assistant that answers from a known domain
• A team with three audio vendors and three bills who wants one
• Anyone measuring time to first audio and finding it too long

Architecture

System overview

Every component, grouped by who runs it. The product sits at the generation call; everything in your lanes is yours and unchanged by this design.

One request, end to end

One key, three endpoints. The audio models are a policy decision, the answer is a routing decision, and every call returns its model and its cost.

Deployment

Where each part runs, and what talks to what over the network.

What stays in your stack

• Audio capture and playback
• Turn-taking and interruption handling
• The conversation state between turns
• Telephony or WebRTC

Technology stack

One choice per layer, the alternative that was considered, and the reason. Swap any row; the Inferbase call does not change.

What carries it

The Inferbase features this design rests on. Each exists in the product today and is linked to its documentation.

Transcription on the same key

A multipart request in the OpenAI shape. Auto takes the first transcription model in your pool; verbose_json returns the duration and segments the provider stated. Documentation

The answer routed on latency

The chat call sends optimize latency, extended_thinking false and a small output cap. A reasoning model is not asked to reason before it speaks. Documentation

Speech streamed as it is made

stream_format audio with response_format pcm returns bytes before the whole sentence is synthesised. Which formats stream early depends on the provider, and the documentation says what was measured. Documentation

The pool decides the audio models

No endpoint routes audio models, because nothing measures them against each other. Your policy pool, in your order, says which transcription and which speech model auto takes, and the editor names them. Documentation

Session affinity across turns

One session identifier per call keeps the answer model steady across turns unless it cannot serve or something beats it by the objective margin, and groups the turns under one run. Documentation

A receipt on every call

Transcription records seconds of audio, speech records characters, the answer records tokens; each with the model that served and the cost. Speech carries its receipt in a header, because its body is the audio. Documentation

Tools on the answer

The answer call may declare tools, so a spoken request can look an order up or book a slot; routing then keeps tool-capable models and the loop stays on one model. Documentation

Fail-over before the first byte

A speech stream can fail over to another provider only before its first byte; a provider that refuses outright falls through to the next in the pool. Documentation

Build it step by step

Seven steps from an empty service to a browser you can talk to. The code is Python for the service and JavaScript for the client; the three-call turn alone is also given in Python and TypeScript after the policy, and both were run against the API on a spoken question. Each step ends with the thing it turns on.

Before you start

• An Inferbase account, a project, and a key with a spend cap for this service.
• Python 3.12 with openai and FastAPI; Redis; a browser with microphone access for the client.
• A policy pool that holds one transcription model and one speech model; the editor names which auto will use.

from fastapi import FastAPI, UploadFile, Form
from fastapi.responses import StreamingResponse
from openai import OpenAI

client = OpenAI(api_key="inf_your_api_key", base_url="https://api.inferbase.ai/api/v1/inference")
app = FastAPI()


def transcribe(upload: UploadFile) -> tuple[str, dict]:
    transcript = client.audio.transcriptions.create(
        model="auto",  # the first transcription model in the policy pool
        file=(upload.filename, upload.file, upload.content_type),  # keep the extension: the provider reads the container from it
        response_format="verbose_json",
    )
    receipt = transcript.model_extra.get("usage", {})
    return transcript.text, {"model": receipt.get("model"), "seconds": getattr(transcript, "duration", None), "cost_usd": receipt.get("cost_micros", 0) / 1e6}

SYSTEM = (
    "You are a phone agent for a store. Answer in one or two short spoken sentences, in words only, never JSON or code. "
    "If you do not know the status of an order, say so and offer to look it up."
)


def answer(call_id: str, history: list[dict], text: str):
    """Yields text chunks; the caller collects them and reads the receipt from the closing event."""
    stream = client.chat.completions.create(
        model="auto",
        stream=True,
        max_tokens=120,
        messages=[{"role": "system", "content": SYSTEM}, *history, {"role": "user", "content": text}],
        extra_body={"routing": {"optimize": "latency", "session": call_id}, "extended_thinking": False},
    )
    for chunk in stream:
        raw = chunk.model_dump()
        if raw.get("object") == "routing":
            yield {"type": "routing", "model": raw.get("model")}
        elif raw.get("object") == "usage":
            u = raw.get("usage") or raw
            yield {"type": "usage", "cost_usd": u.get("cost_micros", 0) / 1e6, "latency_ms": u.get("latency_ms"), "ttft_ms": u.get("ttft_ms")}
        elif raw.get("choices") and raw["choices"][0].get("delta", {}).get("content"):
            yield {"type": "delta", "text": raw["choices"][0]["delta"]["content"]}

def speak(text: str, voice: str = "alloy"):
    """Yields PCM chunks as the provider makes them."""
    with client.audio.speech.with_streaming_response.create(
        model="auto",  # the first speech model in the policy pool
        input=text,
        voice=voice,
        response_format="pcm",  # streams early on the measured provider; other formats are synthesised whole
        extra_body={"stream_format": "audio"},
    ) as speech:
        model = speech.headers.get("x-inferbase-model")
        yield {"type": "speech_model", "model": model}
        for chunk in speech.iter_bytes(chunk_size=4096):
            yield chunk

import json

BOUNDARY = b"\n--audio--\n"


@app.post("/turn")
async def turn(call_id: str = Form(...), audio: UploadFile = ...):
    text, asr_receipt = transcribe(audio)
    history = load_history(call_id)  # Redis, keyed by call id

    def body():
        yield json.dumps({"type": "transcript", "text": text, **asr_receipt}).encode() + b"\n"
        answer_text = ""
        for event in answer(call_id, history, text):
            if event["type"] == "delta":
                answer_text += event["text"]
            yield json.dumps(event).encode() + b"\n"
        yield BOUNDARY
        for item in speak(answer_text):
            if isinstance(item, dict):
                continue  # the speech model name; log it with the receipts
            yield item
        save_history(call_id, history + [{"role": "user", "content": text}, {"role": "assistant", "content": answer_text}])

    return StreamingResponse(body(), media_type="application/octet-stream")

const ctx = new AudioContext({ sampleRate: 24000 })
let nextStart = 0

function playPcm16(bytes) {
  const samples = new Int16Array(bytes.buffer, bytes.byteOffset, Math.floor(bytes.byteLength / 2))
  const buffer = ctx.createBuffer(1, samples.length, 24000)
  const channel = buffer.getChannelData(0)
  for (let i = 0; i < samples.length; i++) channel[i] = samples[i] / 32768
  const source = ctx.createBufferSource()
  source.buffer = buffer
  source.connect(ctx.destination)
  nextStart = Math.max(nextStart, ctx.currentTime)
  source.start(nextStart) // gapless: each chunk starts where the last one ends
  nextStart += buffer.duration
}

async function sendTurn(blob, callId) {
  const form = new FormData()
  form.append('call_id', callId)
  form.append('audio', blob, 'turn.webm')
  const response = await fetch('/turn', { method: 'POST', body: form })
  const reader = response.body.getReader()
  const boundary = new TextEncoder().encode('\n--audio--\n')
  let head = new Uint8Array(0), inAudio = false
  for (;;) {
    const { value, done } = await reader.read()
    if (done) break
    if (inAudio) { playPcm16(value); continue }
    head = concat(head, value)
    const at = indexOf(head, boundary)
    if (at >= 0) {
      for (const line of new TextDecoder().decode(head.slice(0, at)).split('\n').filter(Boolean)) showEvent(JSON.parse(line))
      inAudio = true
      playPcm16(head.slice(at + boundary.length))
    }
  }
}

LOOKUP = {"type": "function", "function": {"name": "lookup_order", "description": "Fetch an order by id.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}}


def answer_with_tools(call_id: str, history: list[dict], text: str) -> str:
    messages = [{"role": "system", "content": SYSTEM}, *history, {"role": "user", "content": text}]
    for _ in range(2):  # a phone call cannot wait for a long loop
        step = client.chat.completions.create(
            model="auto", max_tokens=120, messages=messages, tools=[LOOKUP],
            extra_body={"routing": {"optimize": "latency", "session": call_id}, "extended_thinking": False},
        )
        choice = step.choices[0]
        messages.append(choice.message)
        if choice.finish_reason != "tool_calls":
            return choice.message.content or ""
        for call in choice.message.tool_calls:
            messages.append({"role": "tool", "tool_call_id": call.id, "content": orders.lookup(**json.loads(call.function.arguments))})
    return "Let me check that and call you back."

The routing policy

A chosen pool with one transcription model and one speech model at the top of their types, fast chat models beside them, latency as the objective, and a session cap sized to a call.

Create it under Policies in the portal and attach it to the key the application uses; the steps are in Create a routing policy.

The call

One turn: transcribe the caller audio, answer in one or two spoken sentences, and synthesise the answer with streaming so playback can start on the first bytes. The answer file is raw PCM, which is what a telephony stack plays. The build above wraps this in a service and a browser client.

from openai import OpenAI

client = OpenAI(api_key="inf_your_api_key", base_url="https://api.inferbase.ai/api/v1/inference")

CALL = "call-5521"  # one session id per call: the answer model stays steady across turns

# 1. Transcribe the caller's turn. Auto takes the first transcription model in your pool.
with open("question.wav", "rb") as audio:
    transcript = client.audio.transcriptions.create(model="auto", file=audio, response_format="verbose_json")
print("heard:", transcript.text)

# 2. Answer: short, fast, no thinking.
completion = client.chat.completions.create(
    model="auto",
    max_tokens=120,
    messages=[
        {"role": "system", "content": "You are a phone agent for a store. Answer in one or two short spoken sentences, in words only, never JSON or code. If you do not know the status of an order, say so and offer to look it up."},
        {"role": "user", "content": transcript.text},
    ],
    extra_body={"routing": {"optimize": "latency", "session": CALL}, "extended_thinking": False},
)
answer = completion.choices[0].message.content or ""
print("answer:", answer, f'({completion.model}, {completion.usage.model_extra["latency_ms"]} ms)')

# 3. Speak it, streamed: PCM bytes arrive before the whole sentence is synthesised.
with client.audio.speech.with_streaming_response.create(
    model="auto",
    input=answer,
    voice="alloy",
    response_format="pcm",
    extra_body={"stream_format": "audio"},
) as speech:
    print("speech model:", speech.headers.get("x-inferbase-model"))
    speech.stream_to_file("answer.pcm")  # play as it arrives in production

Both snippets were run against api.inferbase.ai on 2026-10-13 on a spoken question synthesised through the same API: the transcript came back word for word, the answer was served by MiniMax M3 in 2 to 4 seconds, and the streamed PCM landed in a file while it was still being made.

Open in PlaygroundAPI reference

Security and privacy

Voice carries the caller's voice itself, which is personal data, and the conversation. The controls are about what is kept and what a call may spend.

Audio is not kept by the service

The service transcribes the blob and discards it; the transcript is the state. If audio must be kept, that is your store and your retention policy.

Request content at Inferbase

The transcript and the answer are request content, kept for up to 30 days unless zero retention is on for the key; audio sent for transcription and synthesised speech are not stored as request content. The decision record is kept without content.

The key

One key for the service, with a spend cap and a rate limit, never in the browser. The client talks to your service only.

Spend per call

The policy session cap bounds a call; the key's daily cap bounds the day. Speech is charged per character sent, so short answers are the cheapest control.

Tools with consequences

A spoken agent that can act should confirm aloud before a write: "Shall I change the address to this?" The tools guide's approval gate applies, with the caller as the approver.

Injection by voice

Rare, but a caller can read an instruction aloud. The guardrail injection check in record mode shows whether it happens; refuse it once you have seen the verdicts.

Evaluation

Three numbers per utterance, measured by scripts: how well it was heard, whether the answer was right, and how long the caller waited for the first sound. A set of recorded utterances with expected transcripts and expected answers drives all three.

Word error rate

The transcript against the expected transcript, per utterance and by accent or noise condition. It tells you whether the first transcription model in your pool is the right one for your callers.

Answer correctness

The answer against expected content, with a string check or a judge call. Short answers are easy to check; a wrong refusal is the usual failure.

Time to first audio

Measured by the client from the end of the utterance to the first PCM chunk. It is the sum of the three calls' first-byte times; the receipts break it down.

Cost per turn

From the three receipts: seconds, tokens, characters. Compare speech models by characters per second of audio produced as well as price.

Auto against pinned

The answer call pinned to one model against auto on latency: correctness and TTFT. The audio models are a pool choice; compare them by reordering the pool and rerunning.

import time

for case in utterances:  # {"file": "q-017.wav", "expect_text": "...", "expect_any": ["shipped", "look it up"]}
    t0 = time.perf_counter()
    with open(case["file"], "rb") as f:
        heard = client.audio.transcriptions.create(model="auto", file=f, response_format="verbose_json").text
    answer = answer_text(heard)
    first_audio = None
    with client.audio.speech.with_streaming_response.create(model="auto", input=answer, voice="alloy", response_format="pcm", extra_body={"stream_format": "audio"}) as speech:
        for _ in speech.iter_bytes(chunk_size=4096):
            first_audio = time.perf_counter() - t0
            break
    print(case["file"], f"wer={wer(heard, case['expect_text']):.2f}", "ok" if any(s in answer.lower() for s in case["expect_any"]) else "WRONG", f"first audio {first_audio:.2f}s")

Operations

A call is a session, so Usage shows it as one run with three receipts per turn. Watch the three first-byte times, because the caller does.

Time to first audio by component

Transcription latency, answer TTFT, speech first byte, from the receipts and your client. The slowest component is where to spend effort; on the measured pool it is speech, which the streaming form halves.

Which audio models serve

The receipt names the model on every call. A reordered pool changes it on the next call; watch the names after a policy change.

Cost per call

Sum the three receipts per session. Speech characters dominate when answers run long; the output cap and the prompt keep them short.

Fail-over on speech

A stream can fail over only before its first byte. A speech provider refusing outright shows as a fallback in the receipt's attempts; one failing mid-stream shows as a short answer, and your client should play what it has.

Session hops

The routing event on the answer says whether the call stayed on its model. Hops on a phone call are audible as a change of style; a task rule steadies them.

Policy changes

The pool order and the objective have a revision history. Reorder with the evaluation in hand.

Scaling and limits

Calls are concurrent and each turn streams, so the service is long-lived and scales by instance. The limits are the key, file sizes, and the speech provider's streaming behaviour.

Real time, measured: not yet a conversation

A natural conversation needs the first sound inside about 1.5 s of the caller's last word. Measured on 2026-10-14, three turns on a 9 s spoken question: transcription 1.5 to 3.3 s, the answer's first token 2.7 to 5.0 s, its first sentence 3.3 to 5.9 s, the first audio of the reply 1.8 to 3.7 s; the turn, pipelined on the first sentence, 6.7 to 13 s. Routing is 20 to 50 ms of that; the rest is three provider hops. So this design is a turn-based voice assistant today, an IVR, an in-app helper, a voicemail that answers, and not a real-time phone agent. Four things close the gap, in order of payoff: streaming transcription so the answer starts before the caller finishes; a chat route with a sub-second first token winning the latency objective; sentence-level pipelining into speech, which step 5 sketches; and a speech model whose first byte is reliably near the 1.2 s the documentation measured. With all four, expect 2 to 3 s, acceptable for an assistant, still not a conversation.

Concurrent calls

Each turn is three requests over a few seconds. A key at 600 requests a minute carries dozens of concurrent calls; raise the limit before launch.

Utterance size

Transcription takes up to 25 MB per file; a turn is far below it. For a continuous stream, cut utterances at silence on the client or with server-side VAD.

State

The transcript per call id in Redis lets any instance take the next turn. Expire it when the call ends.

Speech formats

PCM streams on the measured provider; a container format may be synthesised whole first. Measure each speech model in the pool before choosing it.

Regions

The three hops are provider latency the service cannot remove. Place the service near the callers, and keep answers short so the speech hop is short.

Telephony

A bridge that sends audio frames needs server-side VAD and a continuous session; the three calls and the receipts do not change.

Troubleshooting

What goes wrong when this is built, what it means, and what to do. Each one was met while building or running it.

Extending

The service sees utterances and returns audio; everything around it can change.

Telephony

A SIP or CPaaS bridge that sends audio frames and plays PCM. Add server-side voice activity detection to cut utterances, and keep the call id as the session.

Interruptions

Stop playback when the caller speaks and start a new turn; the speech request is already paid for, so keep answers short.

A knowledge base

Retrieve passages before the answer call, as the RAG guide does, and keep the answer to two spoken sentences with a citation the agent can read out.

Actions with approval

Confirm aloud before a write, then execute; the tools guide's gate applies with the caller as the approver.

Voices

A voice-design model takes a description instead of a name; keep the voice per deployment, not per turn, so the agent sounds the same across a call.

Your own audio vendors

Connect a provider key under Connections and put its models first of their type in the pool; the receipt records a zero cost and the provider bills you.

What was measured

A full voice turn on the public API, stage by stage, from the caller's last word to the first sound of the answer. The last row is the speech endpoint's own documented measurement on a shorter text; the three runs above are what a turn sees end to end.

Measured from our own infrastructure in India against api.inferbase.ai on 2026-10-14, three runs each, client-side wall time including the network hop (about 0.25 s each way), thinking off. Every response returns its own figures.

Further reading

• The Inferbase quickstarts repository; the voice-agent folder for this guide is planned
• Speech and transcription on the API
• The pool and what auto means per endpoint
• The routing object on a request
• Tools and MCP, for actions on the answer
• Playground: speech and transcription
• Build guide: RAG, for a knowledge base behind the answer

Frequently asked questions