RAG over a knowledge base

Retrieval stays in your stack; the generation call goes to Inferbase with the model set to auto, routed per question on how hard it is, with the context window sized to your retrieval and the privacy decision made where the data flows. Built from scratch on Postgres with pgvector, with the agentic shape included.

What you build

A question-answering service over a knowledge base of your documents: an ingestion job that chunks and embeds them into Postgres with pgvector, a retrieval step that searches, reranks and assembles a prompt, and a generation call to Inferbase with the model set to auto. Each answer cites the passages it used and carries the routing decision and the cost. The same service grows into the agentic shape, where a model decides what to retrieve, without moving the boundary. Embeddings and rerank run through the same key as generation, so the service holds one credential.

When you are done

• An ingestion job that turns documents into cited, searchable passages, with the embedding model pinned
• A retrieval step with vector search and a rerank, through the same API as generation
• A generation call routed per question, with the context window sized from your retrieval and a session that keeps a conversation on one model
• The decision record and the cost beside every answer, read from the response
• An evaluation that measures retrieval recall and answer faithfulness, and compares auto with a pinned model on cost and quality

The components

Ingestion job

Reads documents, splits them at headings and paragraphs into passages of about 1,200 characters with a small overlap, embeds them in batches and writes them to Postgres with their source URL and heading.

Passage store

Postgres with the pgvector extension: one table of passages with a 1,024-dimension vector column and an HNSW index for approximate nearest-neighbour search.

Retrieval

Embeds the question with the pinned model, takes the 20 nearest passages from pgvector, reranks them through the rerank endpoint, and keeps the best 6.

Prompt assembly

Numbers the passages, places them in the system prompt with the citation rules, and sizes the context window the request will require.

Generation

One chat completion with the model set to auto, the routing object carrying the context floor and the session, and thinking off. The response carries the decision and the receipt.

Inferbase

The key, the routing policy with its rule pool and objective, and the three endpoints the service calls: embeddings, rerank and chat completions.

Service

A FastAPI application with one endpoint that takes a question and a conversation id and streams the answer with its sources and receipt.

Evaluation

A labelled question set: for each question, the passages that answer it. Scripts for recall at k, for answer faithfulness through a judge call, and for the auto-versus-pinned comparison.

The problem

In a RAG pipeline the retrieval is the same for every question, and the generation is not.

Retrieval-augmented generation has two halves. Retrieval finds the passages: an embedding, a vector search, perhaps a rerank. Generation reads them and writes the answer. The first half costs about the same for every question. The second half does not: reconciling one retrieved fact with the question is easy, and reasoning across six passages that partly disagree is not.

A pipeline that sends every generation call to the same model pays the hard-question price on the easy ones, or answers the hard ones with a model that cannot hold six passages in mind. The spread between those two is where the money is, and it is invisible when the model is a constant in the code.

The privacy question lives at the same step. Retrieved passages carry whatever the documents carry, names and account numbers included, and the generation call is where they leave your stack. Which models may see them, and whether the request is kept, has to be decided there.

This guide changes one thing in an existing pipeline, and builds the rest from scratch for a new one. Retrieval, chunking, the vector store and the prompt assembly stay exactly where they are. The generation call is sent to Inferbase with the model set to auto, so each question is classified, the context window is checked against what retrieval produced, and the answer is served by the model that wins on your objective, with the decision returned beside it.

Who has it

• A team with a RAG pipeline in production and one model hard-coded at the generation step
• A knowledge base whose questions range from lookups to multi-document reasoning
• Anyone who has to say which model saw a retrieved passage, and keep that record

Architecture

System overview

Every component, grouped by who runs it. The product sits at the generation call; everything in your lanes is yours and unchanged by this design.

One request, end to end

The boundary sits between prompt assembly and generation. Nothing about retrieval changes; the model becomes a decision per question instead of a constant.

Deployment

Where each part runs, and what talks to what over the network.

What stays in your stack

• Chunking, embeddings and the vector store
• Rerank, if you use one
• Prompt assembly and the citation format
• Evaluation of answer quality

Technology stack

One choice per layer, the alternative that was considered, and the reason. Swap any row; the Inferbase call does not change.

What carries it

The Inferbase features this design rests on. Each exists in the product today and is linked to its documentation.

Routing on task and complexity

Each generation call is classified before a model is chosen. A lookup and a six-passage synthesis are different tasks to the router, and they land on different models. Documentation

min_context_window on the request

Size it from your retrieval: chunks times chunk size plus the question and the answer. A model whose window is too small, or not confirmed, is excluded before ranking. Documentation

Session affinity

A conversation id sent as the routing session keeps follow-up questions on the model that answered, unless it can no longer serve or another model wins by more than the margin, and groups the turns under one run. Documentation

Prompt caching where the prefix repeats

A long system prompt or a stable set of passages can be marked for the provider cache. The decision says whether the preference was met and how the chosen provider caches. Documentation

Embeddings and rerank on the same key

The embeddings endpoint takes up to 512 strings a call; the rerank endpoint, in Cohere's shape, scores up to 1,000 documents a call. Auto takes the first model of each type in your pool, so the index is built by one model deliberately. Documentation

The decision record

The response id reads back the decision: classification, the eligible pool, the fallback chain and the per-step trail. The record is kept separately from the request content. Documentation

Zero retention and a pool you choose

Retrieved passages transit the generation prompt. Request content is kept for up to 30 days unless zero retention is on, and the policy pool decides which models may see it. Documentation

Tool-capable routing for the agentic shape

When the model drives retrieval through a search tool, routing keeps only models with confirmed tool support, and the loop stays on the model that opened it. Documentation

Build it step by step

Eight steps from an empty database to a cited, routed answer, then the agentic shape on top. The code is Python throughout; the generation call is also given in TypeScript in the section after the policy. Each step ends with the thing it turns on.

Before you start

• An Inferbase account, a project, and a key with a spend cap for this service.
• Postgres 15 or later with the pgvector extension (any managed Postgres offers it).
• Python 3.12, the openai package, psycopg or SQLAlchemy, and FastAPI. Your documents as text: Markdown, HTML stripped to text, or PDF text layers.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE passages (
  id          bigserial PRIMARY KEY,
  source_url  text NOT NULL,
  heading     text NOT NULL,
  position    int  NOT NULL,          -- order within the source
  text        text NOT NULL,
  embedding   vector(1024) NOT NULL,  -- the pinned model's dimension
  model       text NOT NULL,          -- the embedding model that made the vector
  updated_at  timestamptz NOT NULL DEFAULT now()
);

-- After the bulk load. Cosine distance, so vectors need not be normalised.
CREATE INDEX passages_embedding_idx ON passages USING hnsw (embedding vector_cosine_ops);

import re
from dataclasses import dataclass

@dataclass
class Passage:
    source_url: str
    heading: str
    position: int
    text: str

HEADING = re.compile(r"^#{1,6}\s+(.*)$")
SKIP = re.compile(r"^(related|see also|next steps|further reading)$", re.I)


def chunk(source_url: str, title: str, markdown: str, max_chars: int = 1200, overlap: int = 150) -> list[Passage]:
    sections: list[tuple[str, list[str]]] = [(title, [])]
    for line in markdown.splitlines():
        m = HEADING.match(line)
        if m:
            sections.append((f"{title} > {m.group(1).strip()}", []))
        else:
            sections[-1][1].append(line)

    passages: list[Passage] = []
    for heading, lines in sections:
        if SKIP.match(heading.split(" > ")[-1]):
            continue
        paragraphs = [p.strip() for p in "\n".join(lines).split("\n\n") if p.strip()]
        current, tail = "", ""
        for para in paragraphs:
            if current and len(current) + len(para) > max_chars:
                passages.append(Passage(source_url, heading, len(passages), current.strip()))
                tail = current[-overlap:]
                current = tail + "\n" + para
            else:
                current = (current + "\n\n" + para).strip()
        if current.strip():
            passages.append(Passage(source_url, heading, len(passages), current.strip()))
    return passages


def embedding_text(p: Passage) -> str:
    """What is embedded: the heading first, so the vector carries where the passage sits."""
    return f"{p.heading}\n{p.text}"

import time
from openai import OpenAI, RateLimitError

client = OpenAI(api_key="inf_your_api_key", base_url="https://api.inferbase.ai/api/v1/inference")


def embed_all(texts: list[str], model: str = "auto", batch: int = 128) -> tuple[str, list[list[float]]]:
    """Embed every text; returns the model that did it (pinned after the first batch) and the vectors."""
    vectors: list[list[float]] = []
    for i in range(0, len(texts), batch):
        chunk = texts[i : i + batch]
        for attempt in range(4):
            try:
                result = client.embeddings.create(model=model, input=chunk)
                break
            except RateLimitError:
                time.sleep(2 ** attempt)  # a single-provider model refuses bursts; back off, do not fail the job
        else:
            raise RuntimeError("embeddings kept refusing")
        model = result.model  # pin: every later batch and every query uses this model
        vectors.extend(item.embedding for item in result.data)
    return model, vectors

import httpx
import psycopg

BASE = "https://api.inferbase.ai/api/v1/inference"
HEADERS = {"Authorization": "Bearer inf_your_api_key"}


def query_text(model: str, question: str) -> str:
    if "qwen3-embedding" in model:
        return f"Instruct: Given a question, retrieve knowledge base passages that answer it\nQuery: {question}"
    return question


def retrieve(conn: psycopg.Connection, model: str, question: str, k: int = 6, candidates: int = 20) -> list[dict]:
    vector = client.embeddings.create(model=model, input=query_text(model, question)).data[0].embedding
    rows = conn.execute(
        "SELECT id, source_url, heading, text FROM passages "
        "ORDER BY embedding <=> %s::vector LIMIT %s",
        (str(vector), candidates),
    ).fetchall()
    documents = [f"{heading}\n{text}" for _, _, heading, text in rows]
    reranked = httpx.post(
        f"{BASE}/rerank",
        headers=HEADERS,
        json={"model": "auto", "query": question, "documents": documents, "top_n": k},
        timeout=30,
    ).json()
    kept = []
    for n, result in enumerate(reranked["results"], start=1):
        _, url, heading, text = rows[result["index"]]
        kept.append({"n": n, "url": url, "heading": heading, "text": text, "score": result["relevance_score"]})
    return kept

RULES = (
    "Answer from the passages below and cite them as [1], [2] after the sentence they support. "
    "If the passages conflict, say which applies and why. If they do not settle the question, say so."
)


def build_prompt(passages: list[dict]) -> str:
    numbered = "\n\n".join(f"[{p['n']}] {p['heading']}\n{p['url']}\n{p['text']}" for p in passages)
    return f"{RULES}\n\nPassages:\n\n{numbered}"


def context_floor(system_prompt: str, history: list[dict], max_answer_tokens: int) -> int:
    chars = len(system_prompt) + sum(len(m["content"]) for m in history)
    estimate = chars // 4 + max_answer_tokens
    return max(8_000, int(estimate * 1.5))  # headroom; the floor is a filter, not a budget

def answer(passages: list[dict], history: list[dict], conversation_id: str, max_answer_tokens: int = 500) -> dict:
    system_prompt = build_prompt(passages)
    completion = client.chat.completions.create(
        model="auto",
        max_tokens=max_answer_tokens,
        messages=[{"role": "system", "content": system_prompt}, *history],
        extra_body={
            "routing": {
                "min_context_window": context_floor(system_prompt, history, max_answer_tokens),
                "session": conversation_id,  # follow-ups stay on the same model
            },
            "extended_thinking": False,
        },
    )
    routing = completion.model_extra["routing"]
    usage = completion.usage.model_extra
    return {
        "request_id": completion.id,  # reads back the full decision record
        "answer": completion.choices[0].message.content or "",
        "sources": [{"n": p["n"], "url": p["url"], "heading": p["heading"]} for p in passages],
        "model": routing["model"],
        "task": routing["task"],
        "complexity": routing["complexity"],
        "routing_time_ms": routing["routing_time_ms"],
        "cost_usd": usage["cost_micros"] / 1e6,
        "latency_ms": usage["latency_ms"],
    }

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()


@app.post("/ask")
def ask(body: dict):
    passages = retrieve(conn, EMBEDDING_MODEL, body["question"])
    system_prompt = build_prompt(passages)
    history = body["history"] + [{"role": "user", "content": body["question"]}]

    def events():
        yield json.dumps({"type": "sources", "sources": passages}) + "\n"
        stream = client.chat.completions.create(
            model="auto",
            stream=True,
            max_tokens=500,
            messages=[{"role": "system", "content": system_prompt}, *history],
            extra_body={"routing": {"min_context_window": context_floor(system_prompt, history, 500), "session": body["conversation_id"]}, "extended_thinking": False},
        )
        for chunk in stream:
            raw = chunk.model_dump()
            if raw.get("object") == "routing":
                yield json.dumps({"type": "routing", **{k: v for k, v in raw.items() if k != "object"}}) + "\n"
            elif raw.get("object") == "usage":
                yield json.dumps({"type": "usage", **(raw.get("usage") or raw)}) + "\n"
            elif raw.get("choices") and raw["choices"][0].get("delta", {}).get("content"):
                yield json.dumps({"type": "delta", "text": raw["choices"][0]["delta"]["content"]}) + "\n"

    return StreamingResponse(events(), media_type="application/x-ndjson")

SEARCH_TOOL = {
    "type": "function",
    "function": {
        "name": "search_knowledge_base",
        "description": "Search the knowledge base. Returns numbered passages with their sources.",
        "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
    },
}


def answer_agentic(question: str, conversation_id: str, max_steps: int = 4) -> dict:
    messages = [
        {"role": "system", "content": "Answer the question. Search the knowledge base as many times as you need, then answer from what you found, citing passages as [n]."},
        {"role": "user", "content": question},
    ]
    for _ in range(max_steps):
        step = client.chat.completions.create(
            model="auto",
            max_tokens=600,
            messages=messages,
            tools=[SEARCH_TOOL],
            extra_body={"routing": {"session": conversation_id}, "extended_thinking": False},
        )
        choice = step.choices[0]
        messages.append(choice.message)
        if choice.finish_reason != "tool_calls":
            return {"answer": choice.message.content, "model": step.model_extra["routing"]["model"], "request_id": step.id}
        for call in choice.message.tool_calls:
            query = json.loads(call.function.arguments)["query"]
            found = retrieve(conn, EMBEDDING_MODEL, query)
            messages.append({"role": "tool", "tool_call_id": call.id, "content": build_prompt(found)})
    return {"answer": "I could not find enough to answer.", "model": None, "request_id": None}

The routing policy

Balanced as the objective, a pool you name or a rule on context window, the embedding and rerank models at the top of their types, prompt caching on, and session affinity so a multi-turn conversation over the same documents stays on one model.

Create it under Policies in the portal and attach it to the key the application uses; the steps are in Create a routing policy.

The call

The generation call with retrieval already done: your passages in the system prompt, the question as the user turn, the context window sized to what retrieval produced, and a session identifier so follow-up questions over the same documents stay on one model. The decision comes back with the answer.

from openai import OpenAI

client = OpenAI(api_key="inf_your_api_key", base_url="https://api.inferbase.ai/api/v1/inference")

# Your retrieval, unchanged: the passages your store returned for the question, numbered.
passages = """[1] Refund policy, section 4
https://example.com/kb/refunds#4
Orders cancelled within 14 days of delivery are refunded in full to the original payment method within 5 business days.

[2] Refund policy, section 7
https://example.com/kb/refunds#7
Custom-engraved items are final sale and excluded from the 14-day window."""

completion = client.chat.completions.create(
    model="auto",
    max_tokens=400,
    messages=[
        {
            "role": "system",
            "content": (
                "Answer from the passages below and cite them as [1], [2]. "
                "If the passages conflict, say which applies and why. "
                "If they do not settle the question, say so.\n\nPassages:\n\n" + passages
            ),
        },
        {"role": "user", "content": "I ordered an engraved watch 10 days ago and want to return it. Can I get a refund?"},
    ],
    extra_body={
        "routing": {
            # Sized from your retrieval: chunks x chunk size + question + answer.
            "min_context_window": 32000,
            # Follow-up questions over the same documents stay on one model.
            "session": "kb-conversation-1842",
        },
        "extended_thinking": False,
    },
)

answer = completion.choices[0].message.content or ""
routing = completion.model_extra["routing"]
usage = completion.usage.model_extra

print(answer)
print(routing["task"], routing["complexity"], "->", routing["model"], f'({routing["routing_time_ms"]} ms to decide)')
print(f'{usage["cost_micros"] / 1e6:.6f} USD, {usage["latency_ms"]} ms; decision record: {completion.id}')

Both snippets were run against api.inferbase.ai on 2026-10-13: the answer applied the engraved-item exception from [2] over the general rule in [1], and the decision named a DeepSeek V4 Flash deployment in 5 to 54 ms.

Open in PlaygroundAPI reference

Security and privacy

The passages are the sensitive thing. They leave your stack at the generation call and nowhere else, so that is where the controls go.

Which models may see a passage

The policy pool. A pool you name is a list you can show an auditor; a rule pool is a rule you can show them. Nothing outside the pool is ever chosen, and a request can narrow the pool but never widen it.

Whether the request is kept

Request content is kept for up to 30 days by default, deletable at any time. Zero retention on the key or the organisation keeps it out of the log entirely; the decision record stays, without the content.

Your own keys

For documents that must not leave a contracted provider, build the pool from a provider key or cloud account you connect. The generation call runs on your credential with the model still chosen per question.

Injection in the documents

A passage that tells the model to ignore its rules is text, and the prompt says so. For documents from outside your organisation, run the guardrail injection check in record mode at the prompt door first and read the verdicts under Usage before turning it to refuse; the classifier also refuses questions about injection.

Personal data in passages

Redact at ingestion what the answer never needs: account numbers, addresses. What is redacted is never embedded, never retrieved, never sent.

The key

One key per service, with a spend cap, held by the service and the ingestion job. Clients call your service, never the API.

Evaluation

Three measurements, cheapest first. Retrieval recall tells you whether the right passages reach the prompt. Faithfulness tells you whether the answer stays inside them. The auto-versus-pinned comparison tells you what routing changed. All three run on one labelled set: questions, and for each the passages that answer it.

Recall at k

For each question, embed, search, rerank, and check whether a labelled passage is among the k kept. Report the fraction. Run it after any change to chunking, the embedding model or the rerank; it costs cents.

Faithfulness

Ask a judge, through Inferbase with a JSON schema, whether each sentence of the answer is supported by the cited passage. The judge call is one more routed request with structured output; log its verdicts beside the answers.

Auto against a pinned model

Run the set twice, once with model auto and once pinned to the model you use today. Compare faithfulness, cost from the receipts, and latency. The decision record on each auto answer says which model answered and why.

Read the decisions

Group the auto run by task and complexity from the routing object. The distribution tells you how much of your traffic is simple, which is where the savings are, and which models are winning.

When to run which

Recall after ingestion changes. Faithfulness after prompt or policy changes. The comparison once, then whenever the pool changes.

cases = [
    {"question": "Can I return an engraved item?", "expect_urls": ["https://example.com/kb/refunds#7"]},
    # ...
]

hits = 0
for case in cases:
    kept = retrieve(conn, EMBEDDING_MODEL, case["question"], k=6)
    if any(p["url"] in case["expect_urls"] for p in kept):
        hits += 1
print(f"recall@6 = {hits / len(cases):.2%}")

Operations

What to watch once it serves, and where each number comes from.

The decision per answer

Store the request id with every answer. When an answer is wrong, read the decision record first: the task, the eligible pool, the filter trail, the fallback chain. Then read the sources the retrieval kept. One of the two explains it.

Cost from the receipts

Sum cost_micros from the responses by task and by model. It is the amount charged, so it matches Usage, and it tells you which questions are expensive.

Routing time

routing_time_ms on every response. It scales with prompt length, so a question with many passages classifies slower than a sentence; read your own distribution.

Stale passages

Re-ingest changed documents on a schedule and delete passages whose source is gone. A cited answer from a deleted page is the failure mode nobody reports.

Session hops

The routing event says whether the session stayed on its model and why it moved. A conversation that moves often is a sign the pool is close on the objective; a rule for the task settles it.

Policy changes

The pool and the objective have a revision history. A change applies to the next request, so make it with the comparison in hand.

Scaling and limits

The limits are the endpoints' batch sizes, the key's rate limit, and the store. Each has a known fix.

Real time, measured

A search-and-answer interface, not autocomplete. Measured on 2026-10-14: embedding the question took 0.6 to 2.5 s, the rerank 0.8 to 2.5 s, and the first token of the answer 1.5 to 3.5 s under the latency objective, so the first words arrive 3 to 8 s after the question, with the answer streaming from there. Routing adds tens of milliseconds. For a chat, stream and show the sources first; for anything that must answer in under a second, cache the retrieval and drop the rerank.

Ingestion throughput

The embeddings endpoint takes 512 strings a call; batches of 128 with a retry on 429 keep a job steady. A single-provider embedding model refuses bursts; keep a second one in the pool for rebuilds.

Rerank size

Up to 1,000 documents a call and about 8,000 characters of query. Twenty candidates is the sweet spot; a hundred costs more and rarely changes the top six.

Requests per minute

A key is 60 requests a minute on the free plan and 600 on Pro; a question is three requests. Raise the limit on the key before launch.

The store

An HNSW index answers in milliseconds to millions of rows. Partition by tenant or by source when one table grows past that, and keep the embedding model per table.

Long documents

A question that needs thirty passages needs a model with the window for them. The context floor handles it per request; the pool rule keeps small-window models out.

Cost at volume

Prompt caching on the policy, a stable system prompt, and the simple questions landing on cheap models are the three levers. The routing object lets a caller cap the price per million tokens per request when a workload needs a hard ceiling.

Troubleshooting

What goes wrong when this is built, what it means, and what to do. Each one was met while building or running it.

Extending

Each extension changes one component and leaves the generation call as it is.

Hybrid search

Add a full-text index in Postgres and merge its top results with the vector results before the rerank. Exact identifiers and part numbers are what vectors miss.

Several knowledge bases

One table per source or tenant, each with its own pinned model, selected by the request. The prompt and the policy are shared.

Answer feedback

A thumbs up or down with the request id gives you labelled answers; the ones marked wrong become faithfulness cases.

Enterprise sources through MCP

For sources behind their own APIs, declare an MCP server under Connections and let the model search it as a tool, governed by the policy's allow-list and declared servers.

Your own keys

Build the pool from a provider key or cloud account. The service does not change: it still sends model auto, and the policy decides whose credential serves.

A chat on your site

The customer support assistant guide is this pipeline with a web frame, a handoff and abuse controls; its quickstart runs on an in-memory index that this store replaces.

What was measured

What a question costs in time on the public API: the three calls in order. The decision is the smallest part; retrieval and the provider's first token are the rest.

Measured from our own infrastructure in India against api.inferbase.ai on 2026-10-14, three runs each, client-side wall time including the network hop (about 0.25 s each way), thinking off. Every response returns its own figures.

Further reading

• The Inferbase quickstarts repository; the rag-knowledge-base folder for this guide is planned
• Guide: model routing for RAG systems
• Guide: model routing for agentic RAG
• The routing object on a request
• Reading a routing decision
• Embeddings and rerank
• Security and privacy policies: retention
• Tool governance, for the agentic shape

Frequently asked questions