Router Redesign, Clean Slate Notes (INTERNAL DRAFT)

Internal draft. This page exists to host the router redesign discussion in blog format for local viewing. It is not a published article and must not ship to production. Frontend deploys are manual (npx vercel --prod --archive=tgz), so the safeguard is the human running that command. Before any frontend deploy that picks up this commit, either delete this file or flip published to false.

How this document is maintained

Two operating rules for this doc:

1. Updated in real time. Every decision that lands in the discussion gets written into this doc in the same turn. The conversation is working memory; this doc is the truth.
2. Visual first. Every component, flow, state machine, and decision tree is rendered as a diagram, workflow, table, or both. Prose elaborates; visuals carry the structure.

Rendering constraint: the blog pipeline is markdown plus GFM, not full MDX. So diagrams take one of two forms.

Tables (GFM) carry taxonomies, error mappings, and decision logs.

Where we are in the doc

┌──────────────────────────────────────────────────────────────────┐
│  STATUS: ALL 7 SECTIONS LOCKED. Document complete.               │
│                                                                  │
│  Promise         LOCKED (D1, D2, D3, P7, P8, P9, P10)            │
│                  P10 motion AMENDED 2026-05-29 (two-track GTM)   │
│                  Tagline: "Your routing brain. Your inference.   │
│                            Verified on your data."               │
│  Caller          DONE                                            │
│                  ✓ Shadow replay (D4-D11)                        │
│                    D8/D10 AMENDED 2026-05-29: shadow = lead-     │
│                    qual tool; sales-led conversion for infra     │
│                  ✓ Live routing API (D12-D17)                    │
│  Engine          DONE                                            │
│                  ✓ D18 latency stance                            │
│                  ✓ E1 classifier (NVIDIA at launch)              │
│                  ✓ E2 preset definitions (Auto-only, 2-dial)     │
│                  ✓ E3 per-task benchmark data (AA-only at launch)│
│                  ✓ E4 latency data (AA + passive logs)           │
│                  ✓ E5 balanced-mode scoring (multi-stage filter) │
│                  ✓ E6 fallback chain (static top-3 + 503)        │
│                  ✓ E7 decision caching (none at launch)          │
│                  ✓ E8 Path 3 evidence (storage + display only)   │
│  Failure modes   LOCKED (F1-F8)                                  │
│  Data model      LOCKED (M1-M8 + naming taxonomy)                │
│  Observability   LOCKED (O1-O7)                                  │
│  Implementation  LOCKED (I1-I7 + V1/V2 bifurcation)              │
└──────────────────────────────────────────────────────────────────┘

Why this document exists

We are redesigning Inferbase's router. Two earlier attempts during 2026-05-28 produced designs that were still anchored to the current code, the current schema, the current knob names, and the prior session's framing. Both were thrown out. This document is the canonical record of the new attempt, which starts from a clean slate.

Clean slate, in this case, means:

• No reference to routing.py, classifier.py, sanity.py, inference_routing_logs, model_routes, optimize_mode, model_pool, or any other existing identifier.
• No inherited seam names from the prior attempt (FeatureExtractor, QualityEstimator, ProviderRanker, Executor were the prior set; they may or may not survive a fresh design, but we do not assume they do).
• No inherited locked decisions (min_quality, alpha, soft floor, cooldown durations, error taxonomy, deadline budget, streaming commit point, no cross-brand content fallover, trace schema). None of these carry forward unless re-derived from first principles.
• No assumption that the new design has to "migrate from" or "shadow against" the current router. The new design is judged on its own merits.

What survives:

• Competitive research facts about how the field routes (see internal memory project_routing_competitive_landscape). Field-level findings only, no prescriptive conclusions.
• The "build once, enhance without refactoring everything" principle. Specifically, the insight that contracts and data models are expensive to change later, while component internals are cheap to swap behind stable contracts. This principle is method, not content, and it does not pre-commit us to any particular set of contracts.

Promise (in discussion)

Draft promise (proposed by Vishal, 2026-05-28)

Inferbase is an inference service provider that gives businesses simple, reliable, and efficient access to LLMs. The core principle is that sending all types of prompts to a single LLM is not efficient. Inferbase chooses models based on the requirement and routes prompts accordingly. The caller picks the optimization objective:

Current inference plane (today): reseller, Together + DeepInfra as upstream. Near-future: self-host popular models when demand justifies the GPU spend. Further future: offer fine-tuning and model optimization as services.

Today's stack at a glance

   caller code
       │
       │  one OpenAI-shaped request
       ▼
   ┌────────────────────────────────────────┐
   │  Inferbase                             │
   │  ┌──────────────────────────────────┐  │
   │  │  routing layer                   │  │
   │  │  (mode = cheapest/fastest/...)   │  │
   │  └────────────┬─────────────────────┘  │
   │               │  picks model + endpoint│
   │               ▼                        │
   │  ┌──────────────────────────────────┐  │
   │  │  upstream call                   │  │
   │  └────────────┬─────────────────────┘  │
   └───────────────┼────────────────────────┘
                   │
        ┌──────────┴──────────┐
        ▼                     ▼
     Together              DeepInfra
     (retail price)        (retail price)

The boxed area is everything we control. The two upstream blocks are everything we depend on and pay retail for.

Open pushbacks on the draft promise

Raised 2026-05-28 by Claude. None resolved yet.

P1. The wedge is not visible. "Inference provider that routes with cheapest/fastest/quality/balanced" describes OpenRouter, LiteLLM, Portkey, Requesty, and AWS Bedrock IPR with minor variations. OpenRouter's auto mode already routes (outsourced to NotDiamond); their :nitro and :floor modifiers already map to fastest/cheapest. They have more models, are bigger, are cheaper. "Why Inferbase over OpenRouter" needs an answer that is not "we route."

P2. Reseller economics are unforgiving. Per the stack diagram above, while we are a pure reseller:

The routing has to do enough work to justify the markup. That is a high bar if the routing is the same four modes everyone else has.

P3. The four modes conflate the dial with the engine. Cheapest/fastest/quality/balanced is the user's OBJECTIVE. The PRODUCT is the engine behind the dial. If the engine is "send to cheapest model meeting a quality bar," that is a one-pager and every aggregator already ships it. If the engine genuinely beats alternatives (lower $/answer at the same quality, or higher quality at the same $), that is a product. The draft promise does not specify which one Inferbase is.

Also: "balanced" will be the default. Defaults take roughly 80% of traffic. So balanced is not a fourth mode, it is THE mode. What balanced does is the actual product question, not the user-facing dial.

Reframe: which company is this?

Two different companies are hiding in the draft promise. They look similar in feature lists but differ in everything that actually matters: go-to-market, pricing model, moat, what to build next.

                    "Together but smarter"
                    ───────────────────────
                          inference
                         ┌─────────┐
                         │         │
                         │ ┌─────┐ │
                         │ │route│ │   routing is a tile inside
                         │ └─────┘ │   the inference product
                         │         │
                         └─────────┘


                "NotDiamond with inference attached"
                ────────────────────────────────────
                         routing
                       ┌──────────┐
                       │          │
                       │          │  inference is the substrate
                       │          │  the routing decisions land on
                       │   ┌────┐ │
                       │   │inf │ │
                       │   └────┘ │
                       └──────────┘

These are not "both yes" at pre-seed. The buyer can tell which one we built. The roadmap divergence past month three is large.

Blocking question

Which version is Inferbase? Or name a third version if the two above miss the actual vision.

Vishal's reaction to the reframe (2026-05-28)

"This is the most important decision that changes the whole business dynamics, and honestly I don't know yet, or I am confused."

Together-route as Vishal reads it: bigger market, covers routing + hosting + optimization + fine-tuning, but enormous capital and slow to enter. "Signal is real but entry is difficult."

NotDiamond-route as Vishal reads it: routing is the product, inference attached. "Pitch: use our router if you are using multiple LLMs for your use case, and by the way you can access those models from us too." Easier to build (pure code, software engineering). Worry: routing becomes commodity long-term, intelligence shifts to cost once OSS quality converges with closed.

Pushbacks on Vishal's reading (2026-05-28)

P4. Together-route as framed underestimates the capital wall. "Bigger market" is true and irrelevant if we cannot compete in it. Together has $400M+ raised; Replicate, Modal, Anyscale, Lepton are also in this space. One Llama-70B endpoint at production uptime is roughly $10K/month of GPU rent + on-call + eng. Five models is $50K+/month before a single dollar of revenue. With $400-500K runway, we die before we have a menu. "Inference" is also not one product: Bedrock wins on AWS lock-in, Together on menu + price, Replicate on non-text, Modal on dev ergonomics, Anyscale on Ray. To go Together-route we have to pick an axis Together is structurally weak on. Which axis?

P5. NotDiamond-route as framed underestimates everything that isn't code. Routing the bytes is two weeks of work. The hard parts are not code:

"Pure code" thinking is what kills routing startups. The router is the easy part; the trust and proof infrastructure IS the company. Also: NotDiamond is 18 months ahead with this exact pitch. Why does anyone pick us over them?

P6. The commoditization timing is partly right but the prediction is incomplete. Models do converge at the prior frontier (OSS catches up to closed-frontier-of-12-months-ago). But the frontier keeps moving (GPT-5/Opus-5 extend; OSS chases). And even at quality parity, latency and cost dispersion across providers stays large (different GPUs, different batching, different geographies), so routing retains economic value indefinitely. What commoditizes is the router CODE, not the router DATA. Whoever owns the most "what we would have done on this prompt" decisions wins. So the long-run shifts from "routing commoditizes" to "router code commoditizes; the data and trust around routing concentrate with whoever moves first."

Third option: the proof engine

Pitch: "Run your production traffic through us in shadow mode. We show you exact cost-per-quality across N models on YOUR data. When you are convinced, flip a switch and we route." Inference is downstream of the proof. The router IS the proof, run repeatedly.

The proof engine inverts the trust problem. Instead of asking the buyer to trust our router, we ask them to verify it with their own data, then trust the conclusion.

   How each option asks the buyer to trust
   ────────────────────────────────────────

   Together-but-smarter
   ────────────────────
     buyer ──"buy inference"──► us         (trust = brand + uptime SLA)

   NotDiamond-attached
   ───────────────────
     buyer ──"believe our router is better"──► us   (trust = case studies)

   Proof engine
   ────────────
     buyer ──"send shadow traffic"──► us
                                       │
                                       ▼
                              numbers on buyer's own data
                                       │
                                       ▼
     buyer ◄──────"now route live"─── us   (trust = self-verified)

Upstream question: who is the buyer in month 12?

The three options serve different buyer profiles. "Businesses" is too vague.

Which buyer is Inferbase building for FIRST in month 12? Together-route serves all four poorly (we lose to incumbents on each). NotDiamond-route works for buyer 3 + enterprise. Proof-engine fits buyer 3 cleanly because they can be convinced by numbers.

Pick the buyer first. The route falls out.

Working answer: routing-first, inference-later (2026-05-28, LOCKED)

Vishal's read of the reframe:

"WRT to buyer profile, your suggestion is fine, which means we build the router, tell the customer to test it either with BYOK or proxying through us and then later get into the Together model so we are evolving from routing to routing + inferencing."

Translated:

• Buyer: Series A+ AI product company with 20-40% LLM COGS (buyer 3 above).
• Path: NotDiamond-route now → Together-route later. Evolve from routing to routing + inference rather than picking one.
• Trial mechanism: BYOK or proxy.

   Month 0                     Month 6                      Month 18+
   ───────                     ────────                     ─────────
   Routing only                Routing                      Routing + inference
   BYOK + proxy trial          + self-serve product         (self-host top N models
   Shadow-replay proof         + paying customers           when volume justifies it)
                               + decision data                Path to FT, optimization
                                                              services after that

This is a NotDiamond-shape company today that becomes a Together-shape company once routing volume justifies the GPU capex. The shape changes; the customer relationship is continuous.

Foundational architectural principle (D3, locked 2026-05-28)

Added by Vishal in the same turn as the working-answer lock:

"Ensure that our backend is rock-solid so adding another inference provider is extremely simple, irrespective whether it is another proxy, AWS/on-prem, us hosting, or anything."

D3. The provider layer is an abstraction, not a hard-coded list.

One internal interface for "inference happens here." Adapters cover at least:

The router treats all of these as fungible candidates with uniform attributes (cost, latency, capability, health, available models). Adding a new provider is drop-in adapter work, not router rewrite.

              router
                │
                ▼
   ┌────────────────────────┐
   │ provider abstraction   │   one uniform interface
   │ (cost, latency, caps,  │
   │  health, models)       │
   └─────────┬──────────────┘
             │ adapters
   ┌─────────┼──────────────┬──────────────┬─────────────┐
   ▼         ▼              ▼              ▼             ▼
 Together  DeepInfra   our self-host    cust AWS    cust on-prem
 (proxy)   (proxy)     (later)          (BYO)       (BYO)

Implications for design downstream:

• The router must never know the specific provider type, only the interface.
• Testing the router uses fake adapters; we never integration-test against Together.
• Adding a new provider should be measurable: target is hours-to-days, not weeks.
• BYO-inference becomes a wedge candidate (#6 below) because the architecture already carries it for free.

Pushbacks on the working answer

P7. The compound is partial, not total. Routing → inference shares customers and brand, not infrastructure costs. Worth being honest about which parts compound and which do not.

Twilio-style evolution (SMS → voice; shared customers + brand, separate cost structure), not Stripe-style (payments → billing; shared infrastructure).

P8. BYOK and proxy are not equivalent. Pick the long-run model now. RESOLVED.

Resolution (Vishal, 2026-05-28): build the system on proxy as the primary mechanism. BYOK becomes a feature added later, not a parallel trial mode at launch. Rationale: cleaner architecture, margin from day one, simpler customer onboarding, no divergent code paths to maintain. The trust mechanism for the trial phase is shadow-replay proof (we route your prompts and show you what we would have picked and what it would have cost, without you switching), not BYOK.

P9. "Later" for inference is not a plan. Name the trigger. RESOLVED.

   Trigger math (rough)
   ────────────────────

   Routing volume per month (gross inference $ passing through us)
   $1M    →  No. GPU spend on 1-2 models > saved margin.
   $3M    →  Maybe. Top 1-2 models pay for themselves IF utilization > 70%.
   $5M    →  Yes. Self-host top 3 models. Gross margin ~20% (reseller) → ~50%.
   $10M+  →  Aggressive. Self-host top 5-10. FT offerings start making sense.

   "When do we add inference?" → revenue trigger, not a date.
   Routing data also tells us WHICH models to host first.

P10. What is the wedge vs NotDiamond? RESOLVED 2026-05-28. Candidate list (expanded after D3 lock):

Observations after D3 was locked:

A. D3 makes #6 nearly free to ship. If the backend supports any-adapter cleanly (D3 commitment), "customer's own inference fleet" is just one more adapter type. Architectural commitment carries it; product cost is near zero. NotDiamond and OpenRouter cannot match this without re-architecting hosted-only assumptions.

B. #2 + #5 + #6 stack into a single positioning.

Routing brain you can plug into your own inference fleet. Every decision is inspectable and replayable. Prove it on your own data before you commit.

Single-line candidate:

"Your routing brain. Your inference. Verified on your data."

vs NotDiamond's implicit pitch ("Our router picks the best model for you"):

Clean for the D2 Together-evolution: when we eventually host inference, our pool becomes one more provider option in the customer's routing config. Never force migration; customer chose to add us as a provider.

Resolution (Vishal, 2026-05-28): wedge = {#2 open + auditable} ∪ {#5 proof-driven} ∪ {#6 BYO-inference}. Single-line positioning locked:

"Your routing brain. Your inference. Verified on your data."

Motion: self-serve + dev-first (#3). Pricing-as-wedge (#1): dropped. Inference-attached (#4): becomes true once D2/P9 trigger fires, not a launch wedge.

Sections to fill in (placeholders)

The document will grow into these sections as the discussion produces decisions. Each section stays empty until the corresponding decision lands. The Promise section above is in active discussion; everything below waits on it.

Caller and surface

Two first-class surfaces, both grounded in the locked wedge:

1. Shadow replay, the trial / proof / verification surface. Trust mechanism for the wedge. Discussed in detail below.
2. Live routing API, the production surface. OpenAI-compatible endpoint. To be filled in after shadow replay design lands.

Shadow replay (in active discussion, 2026-05-28)

What shadow replay is, in plain terms

Shadow = runs in parallel, never touches production traffic. Replay = takes real prompts and asks "what would WE have done with this?"

Combined: take the customer's actual prompts (either past ones from a log, or live ones via an SDK), run them through our router, and show them what we would have picked, what it would have cost, and our predicted quality, without changing anything their users see. The router runs in the shadows; production traffic continues untouched.

  Production traffic flow (unchanged during shadow)
  ─────────────────────────────────────────────────

  customer's user ──► customer's app ──► OpenAI/Anthropic ──► user
                              │
                              │  (copy of prompt)
                              ▼
                          our router
                              │
                              ▼
              "for this prompt I'd have picked
               Llama-3.3-70B on Together for
               $0.0008 instead of GPT-4 for $0.014"
                              │
                              ▼
                  aggregated into report

Airline-autopilot analogy: a new autopilot runs in shadow mode for thousands of hours before it's allowed to fly the plane. It computes what IT would have done at every moment; the human flies. Engineers compare. Only then does the autopilot actually fly. Same logic for routing.

What we claim, and what we don't

Sharpened by Vishal 2026-05-28: quality belongs to the model; the router belongs to the pick. We should never claim otherwise.

The router controls:

• Which model gets the prompt
• From which eligible set
• Optimized for which objective (cost / latency / quality / balanced)

The router does NOT control:

• What tokens the model emits
• Whether those tokens are "good"

A router can make the right pick and get a bad outcome (the model is not capable enough for this prompt class). A router can make the wrong pick and get a lucky good outcome (bigger model muscles through). Separate axes.

This honesty actually sharpens the pitch. NotDiamond's implicit pitch conflates routing decision with output quality. Ours does not:

One scope, claim-gated (revised 2026-05-28 from prior two-scope draft)

Vishal collapsed the earlier scope split: selected-pool and auto should not be separated into MVP-vs-v2. Both live in one product. What separates a "claimable opportunity" from a silent prompt is substantiation, not scope.

The contract:

The router can pick from any model in the catalog (auto-eligible by default; the customer can narrow the eligible set if they want). The report only surfaces a switching opportunity for a model M vs the customer's current Z when we can substantiate:

cost(M) < cost(Z) AND quality(M) ≥ quality(Z) on this prompt class.

If we can prove it, we surface it. If we cannot, we stay quiet on that prompt class. The report does not crash on prompts we cannot reason about; those stay "current setup is fine here."

This opens the sales angle Vishal called out: "are you even using the right model for your task?" Auto-suggested opportunities (where the catalog has a cheaper equivalent model from outside the customer's current set) are the headline of the demo. But the claim is gated on substantiation, never speculative.

Three substantiation paths, ranked by cost to deliver:

Path 2 in plain terms, "outside evidence":

Look up published benchmark scores per task family. If candidate model M scores at least as well as customer's current model Z on this task family, claim equivalence at the catalog level.

Example: customer uses GPT-4o-mini for summarizing support emails. We classify the prompt (task = summarization). We look up benchmark scores per task family: GPT-4o-mini = 84, Llama-3.3-70B = 87, Mistral-Small-3 = 82. Llama clears (87 ≥ 84) at one-third the cost → surface as an opportunity. Mistral does not clear → stay silent.

Path 3 in plain terms, "inside evidence":

Take 50-100 of the customer's actual prompts. Run each through both the current model Z and the candidate model M. Have a strong LLM (the "judge") compare the two outputs and say whether they are equivalent quality, or which is better.

Example: customer's log has 5,000 summarize-email prompts. We pick a stratified sample of 50. For each sampled prompt: send to GPT-4o-mini → answer A; send to Llama-3.3-70B → answer B; send both to a judge (GPT-4) with the original prompt: "are A and B equivalent quality answers? If not, which is better?" Aggregate: "47/50 equivalent, 3/50 GPT-4o-mini preferred" → substantiated equivalence with explicit confidence interval (e.g., 94% ± 6% at 50 samples).

How Path 2 and Path 3 work together (sequential, not alternatives):

Path 3 runs ON TOP of Path 2 candidates, not independently. Sequence per report:

   Step 1: Path 2 surveys broadly
           "Catalog suggests 12 cheaper candidates with equivalent
            benchmarks for your task families."

   Step 2: Path 3 validates narrowly
           "Sample-call validation runs on the top 3 (highest projected
            savings). Sample size 50 each, judge says equivalent on 47/50,
            49/50, 41/50."

   Step 3: Report combines per-opportunity evidence tiers
           Each switching opportunity carries WHATEVER EVIDENCE we have
           for it, Path 2 only, or Path 2 + Path 3 layered.

Evidence tier shown per opportunity in the report:

The bolded line is the upgrade. Same opportunity, stronger evidence. The customer never picks "Path 2 vs Path 3", they see whichever evidence the report has on each opportunity, and Path 3 is automatically applied where it is worth the sample budget (typically the top-N by projected savings).

Path 1 (same-model arbitrage) runs underneath both, always-on, free, dense coverage.

   Coverage of substantiated switch-opportunities per customer
   ──────────────────────────────────────────────────────────►

   Same-model provider arbitrage   ███████████████████  high
   Benchmark-backed equivalence    ████████             moderate
   Sample-call validation          ██                   small (budget-bound)

   Launch: arbitrage is dense, family-equivalence catalog-bound,
           sample-validation tiny.
   Over time: bands 2 and 3 widen as catalog + sample pipeline mature.
              Customer sees the product visibly improve.

Why this is cleaner than the prior MVP-vs-v2 split:

• Customer sees ONE coherent product from day one
• Sales gets the "are you using the right model?" angle from launch
• Reports stay honest from launch, we never claim a saving we cannot substantiate
• Product visibly improves over time as substantiation paths 2 and 3 mature (this is a feature for sales narrative, not a constraint)

Same gating applies in live mode. When the customer flips live, the router routes to M instead of Z only when at least one substantiation path supports it. Otherwise it stays on Z (or the customer's preferred default for that task class). Same honesty contract for shadow and live.

Why we need it

Our buyer (Series A+ AI product company, LLM is 20-40% of COGS) fears two things:

1. Wasting money on overkill models, GPT-4 doing work a Llama could do.
2. Quality regression their users notice, switching to a cheaper model and watching their NPS drop. Worse outcome than paying more.

When any routing vendor (us included) says "we'll save you 30%," they cannot just believe it. Every weaker trust mechanism fails for this buyer:

Shadow replay is the only mechanism that produces a yes/no answer on the customer's own data, without making them change anything in production. It bridges "I'm interested" to "I'll flip live."

This is also the entire mechanism behind the wedge. Without shadow replay, "verified on your data" is a slogan. With shadow replay built well, the slogan IS the product.

  NotDiamond / OpenRouter typical pitch     vs.     Inferbase with shadow replay
  ─────────────────────────────────────             ────────────────────────────
  "Trust us. Here are case studies."                "Run it on your data."
            │                                                  │
            ▼                                                  ▼
     buyer hesitates                              buyer gets proof in days
            │                                                  │
            ▼                                                  ▼
     enterprise sales cycle                         self-serve conversion

Design: two modes of shadow replay, one engine

Q1 in active discussion 2026-05-28. Plain-language breakdown of each mode below.

Mode A, Offline shadow: "upload a batch, get a report"

Customer uploads a file of past prompts (JSON log, CSV, paste sample, or import from provider usage logs). We process server-side, run the router on each prompt, compute substantiations, deliver a report in minutes.

   customer's past prompts (log file)
              │
              │  one-time upload
              ▼
          our backend
              │
              │  router runs on each prompt, predicts savings
              │  Path 3 sample-calls run on top opportunities
              ▼
       report (dashboard / PDF)
              │
              ▼
       customer reads it, decides

Mode B, Online shadow: "continuous observation on live traffic"

Customer integrates a thin SDK call in their code. Their app keeps calling their current provider as normal. We get a copy of each prompt + response, run the router decision in parallel, update a continuous dashboard. No production routing change.

   customer's production code
   ──────────────────────────
       prompt = "..."
       inferbase.observe(prompt, model="gpt-4o-mini", response=their_response)
       # their actual call to OpenAI happens as before, no change

   What inferbase.observe() does:
       - logs the prompt + which model they used + what response came back
       - runs our router decision in parallel
       - sends to dashboard for continuous reporting

   Customer's users see: nothing different. Production untouched.

How they fit together:

   landing page                     "ran a trial on past logs.        live routing
   ────────────                      looks promising, want to          (production)
        │                           validate on real traffic"               │
        ▼                                        │                          │
   OFFLINE shadow                                ▼                          │
   (5-min report)        ───►               ONLINE shadow     ───►          │
                                           (1-2 weeks)                      │
                                                                            │
                                                                       flip live,
                                                                       pay for routing

Two different products doing different jobs in the same funnel. Not alternatives.

Engineering reality:

Note: online shadow at launch is the light SDK-based version (inferbase.observe()), not a full proxy. Full proxy is the "live routing" product, not shadow.

Trade-off for the launch decision:

Q5: the flip-live moment (in active discussion 2026-05-28)

The highest-stakes moment in the customer journey. Before flip: nothing in production changes. After flip: every prompt may be routed to a different model than the customer is used to. A bad first day in live mode could undo months of trust building.

Two ways to handle the moment, on a spectrum:

Option A, Our recommendation drives the decision.

   Report: "Based on your data, we recommend flipping live now.
            Estimated monthly savings: $X."

           [ Flip Live ]

Simpler UX, faster time-to-live. But contradicts the wedge: "Verified on your data" with a button labeled "Trust us, flip now" is a contradiction. This is what NotDiamond already does.

Option B, Customer-defined threshold + auto-fallback safety net.

Customer sets the rules; we execute. Two parts plus an ongoing control surface.

   ┌────────────────────────────────────────────────────────────────┐
   │  Going Live, Configure your routing                           │
   │                                                                │
   │  ○ Conservative                                                │
   │     Apply only opportunities with Path 3 validation ≥45/50.    │
   │     Estimated savings: $42K/year (3 opportunities)             │
   │                                                                │
   │  ● Balanced [recommended default]                              │
   │     Apply opportunities with Path 2+3 OR Path 2 alone when     │
   │     benchmark gap is ≥5 points.                                │
   │     Estimated savings: $87K/year (8 opportunities)             │
   │                                                                │
   │  ○ Aggressive                                                  │
   │     Apply all opportunities with any substantiation.           │
   │     Estimated savings: $112K/year (12 opportunities)           │
   │                                                                │
   │  Custom thresholds: [Advanced settings]                        │
   │                                                                │
   │  Hard rules: [Add per-route pin / per-task block]              │
   └────────────────────────────────────────────────────────────────┘

Customer sees savings shift per risk tolerance. They pick. We provide a recommended default but never make the choice.

Real-time safety net (always on):

Ongoing override controls (beyond the flip moment):

The flip-live moment is the entry to a control surface, not a one-time decision.

• Per-route pin: "always use GPT-4 for this customer-support endpoint"
• Per-request override: HTTP header to force a specific model
• Full model block list: "never route to Mistral"
• Quality drift monitoring: alert if Path 3 re-validation degrades over time

Engineering trade-off:

Claude's push: Option B at launch. Three presets so 80% of customers never touch advanced settings, controls there for the 20% who want them, safety net always on. If we ship A first and add B later, the trust mental model is harder to rebuild than to build right from the start.

Open sub-decisions (Q1-Q6, awaiting Vishal):

Quality prediction cold-start path (relevant to Q4):

   public benchmarks (per-model baseline competence per task type)
                 +
   offline LLM-judge labels on curated eval set (a few thousand prompts)
                 +
   prompt classifier at runtime (task type + difficulty)
                 ↓
        predicted quality with WIDE confidence intervals
                 ↓
   feedback signals accumulate (customer evals, sanity flags, thumbs)
                 ↓
        confidence intervals tighten over time

This is the field's standard bootstrapping path (per the trimmed project_routing_competitive_landscape research). Honest version: early reports show predicted quality with wide CIs that visibly tighten over months. The CI itself becomes the trust signal.

Why shadow replay is the moat, not just the trial:

The cross-customer "what we would have routed" decision log compounds:

• Training data for the quality predictor (tightens CIs over time)
• Industry benchmarks for sales material
• Defensibility against late entrants (they cannot recreate past flows)

Router code commoditizes in 2 weeks. Shadow-replay data accumulated per customer engagement does not. THIS is what makes the wedge defensible long-run.

Live routing API

In active design 2026-05-28.

Endpoint shape

LOCKED: OpenAI-compatible drop-in. Customer changes the base URL; everything else is the same. Maximum compatibility with existing tooling (LangChain, LlamaIndex, the OpenAI SDK in any language, etc.). Anthropic-compatible endpoint is a phase 2 addition; ship with OpenAI compat at launch.

Authentication

LOCKED 2026-05-28: API keys at launch. IAM/SSO is phase 2 once enterprise demand materializes. Dashboard sign-in uses OAuth (Google/GitHub), separate surface from the inference API.

Why not OAuth for the inference API: OAuth is the wrong pattern for service-to-service inference calls. It is for user-mediated flows (sign-in, third-party app authorization). Every inference vendor (OpenAI, Anthropic, Together, NotDiamond, OpenRouter) uses API keys for runtime; OpenAI-compatibility from above forces the same here anyway.

API-key sub-decisions, locked:

The per-key scoping is the runtime side of D10's override controls. A customer can have one key that uses full auto-routing, one locked to a specific model for a critical endpoint, and one with a $100/mo spend cap for a side project. The "customer agency" of D10 applies at the auth layer.

Per-request overrides

LOCKED 2026-05-28 (after multiple revisions; final shape is OpenRouter-style). Multi-key (D13) is the preferred way to express stable routing intent; per-call overrides exist for dynamic intra-service decisions and for migration ergonomics.

Four capabilities:

Why pinning customers exist (value props per case):

Most customers do not pin 100% of traffic. They pin some endpoints (the critical user-facing ones where they trust a specific model) and let other endpoints route (batch jobs, internal tools). Both happen in the same codebase, often using the same key. We capture both.

Multi-key is THE way to express stable routing intent (per D13):

For architectural splits, customer creates multiple keys with different per-key configs:

   Stable architectural splits  →  Multi-key (D13)
   ────────────────────────        ───────────────────────────────
   "batch jobs use cost,            key_batch = "ibk_batch_..."   (auto:cost, all models)
    UI uses quality"                key_ui    = "ibk_ui_..."      (auto:quality, GPT-4 + Claude only)

   Dynamic per-call decisions   →  Per-call mode override
   ────────────────────────        ──────────────────────
   "premium user gets quality,      f"auto:{user.tier}"
    free user gets cost, same
    service, one key"

Decoding rule (precedence, highest wins):

1. Body extension router field (explicit per-call config)
2. Model-name prefix (auto:<mode> or literal model ID)
3. API key's configured default routing mode

Edge cases locked:

Response surface

LOCKED 2026-05-28. Minimal, strictly OpenAI-compatible body. Transparency lives in the dashboard, accessible by request ID.

Response shape:

{
  "id": "chatcmpl-abc",
  "choices": [...],
  "model": "llama-3.3-70b-instruct@together",
  "usage": {...}
}

The standard OpenAI model field is populated honestly with what we picked + which provider served it (e.g., llama-3.3-70b-instruct@together). Not a custom field, the existing OpenAI field, used truthfully. Same pattern OpenRouter uses.

Why minimal:

Vishal's push (2026-05-28): dashboard is the right place for transparency. Engineering teams don't inspect response headers per call; they look at dashboards for system-wide insight. Adding X-Inferbase-* headers + opt-in _inferbase body would be over-engineering for a wedge gesture customers wouldn't use day-to-day. Net: less engineering surface, stricter OpenAI compat, dashboard as canonical source of truth.

Adjacent product implications that flow from this decision:

1. Dashboard must support request-ID lookup. When an engineer sees an unexpected model value in their logs and wants to understand why, pasting the request ID into our dashboard surfaces the full routing decision (alternatives considered, evidence tier, cost breakdown, fallback reason if any). This is a real UX requirement.
2. Power-user escape hatch: routing-decisions API. Separate endpoint, e.g. GET /v1/routing-decisions/{request_id} or GET /v1/routing-decisions?from=...&limit=..., gives programmatic access for customers who want to build their own monitoring, export to Datadog / Sentry, or run audit pipelines. Not bloating inference responses. Available to anyone who wants it.

Streaming

LOCKED 2026-05-28. Standard SSE streaming with first-byte semantics for fallback.

SDK documentation contract: "First-byte semantics. Pre-content failures are transparent; post-content failures bubble up. Same as any single-vendor stream."

Deferred (not launch):

• Cross-model fallback mid-stream (would cause visible content style/quality shifts; bad UX)
• "Skip routing entirely for ultra-low-latency" mode (contradicts the routing value prop; defer until first customer asks)
• Streaming-specific per-key timeouts (defaults should work; revisit if needed)

Pricing surface

LOCKED 2026-05-28. Token markup, no public disclosure of the markup percentage. Standard industry practice; every major inference vendor publishes rates, none disclose their cost structure.

Public pricing page:

   Llama-3.3-70B-Instruct      $0.22 / M input    $0.25 / M output
   GPT-4o                       $3.50 / M input    $13.50 / M output
   Claude-Haiku                 $0.30 / M input    $1.50 / M output
   ...

   Free shadow replay during trial. Live routing prices above apply
   after first paid request.

Monthly invoice:

   Llama-3.3-70B-Instruct       12.4M tokens         $2,983
   GPT-4o                        0.8M tokens          $4,200
   ────────────────────────────────────────────────────────
   Subtotal                                           $7,183

No "includes X% markup" footnote anywhere. Customer sees rates and what they pay; that is the full pricing surface.

Internal launch markup: 25%. Not disclosed publicly. Re-evaluate after first ~50 customers based on conversion data, churn, and customer feedback.

Why we do not disclose the markup:

Earlier Claude draft was to publish "25% markup over upstream" for wedge alignment. Vishal pushed back, correctly: that was conflating product transparency with business transparency. The wedge is product transparency:

• Routing decisions (D9, D15): inspectable
• Evidence (D6, D7): inspectable
• Which model handled the call (D15): visible in model field
• Cost to customer: visible on invoice

It is NOT business transparency. Disclosing our margin gives competitors pricing intelligence and customers a "go direct" thought, without adding any product value to the buyer.

Sales answer to "are you gouging us?" leads with delivered value, not cost structure: "Here is what you pay (transparent on the pricing page). Here is the routing value this month (shown in your dashboard: decisions, alternatives, evidence). Here is the savings vs your prior setup (shadow-replay baseline). Pricing reflects this value; you can verify it on your data."

Sub-decisions locked:

The engine (in active design 2026-05-28)

Inputs and outputs collapsed into this section since they're determined by the Caller and surface decisions above. What's left is the routing pipeline: how a request becomes a routing decision.

High-level pipeline

   request arrives at /v1/chat/completions
      │
      ▼
   API boundary inspects model field
      │
      ├── literal model pin (e.g., model="gpt-4")
      │       │
      │       ▼
      │   COMPAT HANDLER (bypasses the engine)
      │   • Look up model in catalog
      │   • Pick cheapest healthy provider (Path 1)
      │   • Hand off to transport
      │
      └── model="auto" or "auto:<mode>"
              │
              ▼
          ┌────────────────────────────────────────────────────────┐
          │ ENGINE PIPELINE                                        │
          │                                                        │
          │ 1. Resolve effective config (key default + overrides)  │
          │ 2. CLASSIFY prompt (task family, difficulty, caps)     │
          │ 3. FILTER candidates (capability + eligible_pool)      │
          │ 4. APPLY QUALITY FLOOR (per customer's preset)         │
          │ 5. SCORE & RANK by routing mode                        │
          │ 6. PICK + fallback chain (Path 1 for provider)         │
          │ 7. Hand off to transport (D16 semantics)               │
          └────────────────────────────────────────────────────────┘

Substantiation in SHADOW mode vs LIVE mode is the same EVIDENCE but answers a different question: shadow asks "would M be cheaper than the customer's CURRENT default Z, while quality-equivalent?"; live asks "given the customer's chosen routing mode and quality floor preset, is M an acceptable pick?". Paths 1/2/3 supply the evidence in both.

Engine sub-decisions

E1: Classifier (LOCKED 2026-05-28)

Decision: vanilla NVIDIA prompt-task-and-complexity-classifier (ONNX-quantized) at launch as the sole classifier. No fast-path tier at launch. No custom classifier work inside Inferbase scope at launch.

Latency stance (re-anchored from earlier draft): sub-10ms was an arbitrary target. Re-anchored to accuracy-first within ~100-200ms budget. Classification adds 50-100ms, invisible against 200-2000ms LLM TTFT for most use cases. Voice AI and real-time agentic workflows are real edge cases but not the launch target; if they materialize, a fast-path tier (Model2Vec embeddings, NOT regex) is a phase-2 addition.

Options eliminated:

Option E chosen because: pretrained (no cold-start data needed), outputs 11 task categories + 0-1 complexity in one call, MIT-licensed, ~50-100ms with ONNX quantization, already validated in the field.

Moat path (separate from Inferbase launch scope):

Vishal personally takes on fine-tuning or training-from-scratch as an isolated research project, NOT consuming Inferbase engineering bandwidth. Purpose is dual:

1. Optionality: if the research project produces a classifier that beats vanilla NVIDIA on our customer distribution, it becomes a candidate swap-in later
2. Personal capability: the FT/optimization service evolution in D2 needs deep LLM expertise; this research builds it

Caveats: research projects without explicit ship deadlines tend to languish. Vishal sets quarterly self-checkpoints. Success metric is the research goal ("outperforms NVIDIA in benchmarks on at least one task family") rather than the engineering goal ("deployable to Inferbase") to avoid scope pressure.

The anonymized aggregate data (D11) is still collected for other moat reasons regardless of classifier path. If/when the research project produces something good, that data IS the training set.

Sub-decisions inside E1:

E2: Preset definitions (LOCKED 2026-05-28)

Key insight: preset only applies in the Auto routing path. When customer specifies a Custom pool, they have already done the model vetting; preset would be a redundant filter on a pre-filtered set. So preset is scoped to Auto only; Custom-pool customers just pick a mode.

Two-dial design (Path A):

   Pool type
   ─────────

   ○ Custom (customer specifies models)
       Pool: [GPT-4, Claude-Haiku, Llama-3.3-70B]
       Mode: cost / quality / balanced / latency
       (no preset, customer has pre-filtered)

   ● Auto (Inferbase picks from catalog)
       Preset: Strict / Standard / Permissive
       Mode:   cost / quality / balanced / latency

Preset names, renamed from D10's original Conservative/Balanced/Aggressive to avoid "Balanced + balanced" naming collision with the balanced mode:

Quality floor mechanism (launch-placeholder values, to be tuned post-launch):

Numeric thresholds are launch placeholders. Tune within first 30-60 days based on real customer evaluations and feedback. Customer-facing names do not change; numbers behind them do.

Combination matrix (preset × mode, Auto-pool only):

Soft fail when no candidate clears the floor: route to closest-match candidate (most- permissive substantiated option) and log fallback reason to the dashboard routing-decision record (per D15). Hard-fail mode available as a key-level setting for customers who prefer error-over-regression.

Default preset for new customers: Standard.

Path B (named intents like "Max savings") demoted to documentation patterns, not UI elements. Docs and tutorials suggest combinations:

• Cost-sensitive workload → Auto + Permissive + cost
• Quality-critical user-facing → Auto + Strict + quality
• Voice agent → Auto + Strict + latency

Customer-facing dashboard shows two dials explicitly. Long-term reliability (less translation surface, predictable semantics, transparent traceability) wins over first-impression simplicity.

E3: Per-task benchmark data (LOCKED 2026-05-28)

REVISION IN PROGRESS (2026-05-29): the mapping table and metric list below are STALE (they reference humaneval / ifeval / mgsm, which do not exist in our prod AA data, and they mis-assign the QA pair). See the E3 mapping review (2026-05-29) amendment at the end of this E3 section for the corrected metric set, the usage-vs-capability root cause, the Open/Closed QA correction, the empirical prod data, and the verified LiveBench gap-fill option. The scoring function, confidence tiers, and contamination handling below are unchanged and still hold.

What E3 is solving

Path 2 (D7) needs per-(model × task_family) quality scores normalized to 0-1 so E2's quality floors are evaluable per prompt. The catalog today holds aggregate scores via the model_benchmarks table (one row per model × suite; today the only suite is artificial_analysis). E3 is the projection layer from those aggregate scores onto NVIDIA's 11 task families from E1.

E3 produces two values per (model, task_family) cell:

1. quality_score ∈ 0..1 (or NULL if no metrics map to this cell)
2. confidence ∈ {High, Medium, Low, NULL} based on how many metrics support the cell

Data source at launch: catalog only

   model_benchmarks (AA suite, already ingested)
        │
        ▼ projection layer (the work E3 actually adds)
        │
   (model, task_family) → (quality_score, confidence)
        │
        ▼ consumed by
   E2 absolute floor + D6 relative gate + E5 scoring

No new ingest at launch. AA covers most catalog models with 8-12 metrics each (intelligence_index, coding_index, mmlu_pro, gpqa, humaneval, math, ifeval, mgsm, etc.). Models AA does not cover have NULL cells across all task families; for those models, Path 1 same-model arbitrage and Path 3 customer-validation still work. Path 2 stays silent on them, honestly.

The model_benchmarks table is suite-extensible (designed for multiple suites without schema change). Post-launch we review the catalog, identify which AA-missing models matter, and decide how to fill (model card / paper claims as a model_card suite, lm-eval-harness as a lm_eval_harness suite, or targeted custom evals). Not at launch.

Granularity: NVIDIA's 11 task families, with silent cells

Per-task using the 11 categories from E1's classifier output. Silent NULL cells where no metric maps to a family. Aggregate-only was rejected (violates D6's "on this prompt class"). Coarse buckets were rejected (loses code-vs-reasoning differentiation, which is the most intuitive demo case).

Honest cold-start coverage from AA's metric set:

AA's metric set is QA + code heavy. Five families silent at launch. D1 buyer's traffic concentrates in QA / code / generation, so the Path 2 vocal zones cover the headline proof cases; Path 3 picks up the silent families when sample budget allows. The exact mapping table is committed as code, reviewable and amendable as we learn.

Scoring function

Two-stage normalization:

   Step 1: per-metric rank normalization across the catalog
   ──────────────────────────────────────────────────────────
   For each AA metric m:
     score_m_normalized(model) = rank(model, m) / |catalog_with_score_for_m|

   Rank normalization makes metrics with different absolute scales
   (MMLU 0-100, Arena-Hard 0-100 win-rate, GPQA 0-100) comparable.

   Step 2: aggregate per task family
   ──────────────────────────────────
   quality_score(model, task_family) =
     mean(score_m_normalized(model) for m in mapping(task_family))

   Cells with zero metrics: NULL → Path 2 silent.

Equal-weighted mean at launch. No premature opinion on which metric matters more. E8's feedback loop is the place to revisit weights once Path 3 evidence accumulates.

Confidence tiers

Preset interaction:

Dashboard surfaces confidence per opportunity so customers see whether a claim leans on 3 corroborating metrics or just one.

Absolute vs relative interpretation (clarifies E2's floor)

E2's preset floor values 0.85 / 0.70 / 0.50 are on the rank-normalized 0-1 scale where 1.0 = top model in catalog on that task family. Not on a benchmark accuracy scale.

Both must pass. Floor filters the eligible pool; substantiation gates the claim.

Contamination handling

Public benchmarks are partially contaminated; some models have seen the test sets. Two pragmatic decisions:

1. Rank normalization blunts absolute contamination effects. Most popular models are contaminated against MMLU; the relative rank still conveys information.
2. Per-(model, metric) denylist table or column, ready at launch but empty. When the community flags a specific (model, metric) pair as contaminated, populate the denylist and the projection layer excludes that pair from the calculation.

We do not litigate "are public benchmarks meaningful?" in the dashboard. Path 3 is the customer-specific answer; Path 2 is the catalog-level signal.

Silent task families: accepted at launch

Rewriting, Brainstorming, Extraction, Summarization, Text Generation, Chatbot all start NULL or thin. Path 2 stays silent on these for prompts that classify into them. The report shows "current setup is fine here" for those prompts, with no false claim. Path 3 (sample-call validation) fills the gap for customers whose workloads concentrate in these families and who allocate sample budget.

Maintenance and gap-filling

Custom evals (Option C path) are explicitly NOT scheduled at launch. On-demand only, triggered by customer engagement signal.

Sub-decisions locked

Long-term research thread (parked)

Two alternative source strategies stay parked for post-launch evaluation:

• lm-eval-harness as a second suite when AA stops being sufficient or trustworthy
• Curated own + LLM-judge narrowly scoped to silent task families OR if AA + harness still leaves coverage gaps

Both are real options; neither is launch work. The model_benchmarks suite-extensibility keeps both available as drop-in additions when signal warrants.

E3 mapping review (2026-05-29) — findings, provisional decisions, gap-fill research

Status: Part 1 of the E3 review (which benchmark maps to which task family) is under review with Vishal (ML). Decisions below are PROVISIONAL. Part 2 (scoring validity, floor calibration) is not started.

Root cause: usage taxonomy vs capability taxonomy. NVIDIA's 11 families come from the InstructGPT use-case taxonomy (the model card cites the InstructGPT paper). They describe what a user is trying to do, sampled from real API traffic. AA's benchmarks describe what capability a model has (reasoning, coding, math). These are orthogonal framings, which is why there is no clean correspondence. AA cleanly covers only 2 of 11 families.

Correction: Open QA vs Closed QA is not about answer format. NVIDIA's actual definitions (from the model card):

• Open QA = answer from the model's general knowledge, no context provided. ("What is the capital of France?")
• Closed QA = answer from text/data provided in the prompt, i.e. reading comprehension. ("Based on this document, what was Q3 revenue?")

Implication: mmlu_pro / gpqa / hle measure parametric knowledge, so they are Open QA, not Closed QA. Closed QA needs a reading-comprehension benchmark, which our prod AA data does NOT carry. The old table above (Closed QA = mmlu_pro, gpqa, math, mgsm) is wrong on both the metric names and the assignment.

Real prod AA metric set (replaces the stale humaneval / ifeval / mgsm list):

Note the scale split (composites 0-100, individuals 0-1). Rank-normalization neutralizes it because each metric is ranked independently; never average raw scores across them.

Empirical findings (prod, 82 AA rows):

• Composites contain their own components, confirmed by correlation: coding_index vs scicode r=0.88, vs livecodebench r=0.68; math_index vs aime r=0.82, vs math_500 r=0.65. Listing a composite alongside its components triple-counts one signal: it inflates the confidence tier AND over-weights that signal in the mean.
• intelligence_index is a generalist: it correlates 0.75-0.96 with every per-family metric (0.96 with coding_index, 0.90 with hle). As a universal baseline it just re-injects one global ranking and flattens per-family differentiation. Keep it for Other only.
• mmlu_pro vs gpqa r=0.92 (nearly redundant); hle is the independent knowledge signal (0.53 with mmlu_pro).
• AA's coding/math composites are from an OLDER AA schema. AA's current Intelligence Index v4 dropped standalone Coding/Math indices and is a 4-bucket weighted average, so our cached composites reflect an earlier composition.

Coverage + a load-bearing downstream consequence:

• Only 14 of 67 routable models carry any AA data. The projection produces quality cells for those 14 only; the other 53 are silent everywhere and thus unroutable in the Auto pool (acceptable as an explicit stance: Auto routes only among characterized models; the rest stay reachable via Custom pool / pinning).
• Silence = 503. eligibility.py:cell_clears(None, ...) returns False for every preset including Permissive, so a silent family yields an empty pool through all F2 relaxations and the caller returns HTTP 503 no_eligible_candidates. So the silent families are NOT a "fill later" nicety: leaving Summarization / Rewriting / etc. silent means the router hard-fails on those very common prompts. This makes the unmappable families load-bearing.

Provisional Part 1 mapping (3 decisions, pending Vishal sign-off):

Gap-fill research: who covers the families AA misses. AA and LiveBench are almost perfectly complementary (AA = knowledge + code + math capability; LiveBench = the generation/transformation usage tasks):

• LiveBench is viable and the better gap-fill source. 99 of 119 LiveBench models normalize onto a catalog entry; the 20 misses are creators we don't carry (GLM/zhipu, Grok/xAI-paused, arcee/elephant/mimo). Data is a plain GitHub CSV (livebench.github.io/public/table_2026_01_08.csv), refreshed ~monthly, no API key, tracks the frontier within weeks. Caveat: covers the head (~119) not our long tail (529 text LLMs), and joining needs a real model-alias layer (IdentityResolver work).
• HELM is NOT viable at launch. Its scenarios map best conceptually (it was built around this usage taxonomy: summarization, classification, reading-comp QA, extraction), but its model registry is dynamic and its leaderboards lag the frontier by months, so it lacks our current fleet. Keep HELM as reference for scenario design, not as an ingest source.

Roadmap impact: the E3.1 / E3.6 "fill silent families post-launch" item is now concrete: ingest the LiveBench category CSV + build the alias layer to fill Summarization / Rewriting / Text Generation (direct) and corroborate Code/Math. HELM is downgraded to reference-only. Remaining hard gaps with both sources: Chatbot, Classification, Brainstorming.

E3 model-coverage analysis (2026-05-29) — the gap is rows, not columns

We chased the family (column) gap first, but verified prod data shows the dominant gap is the MODEL (row) gap, and it is the cost-relevant open models that are dark.

Coverage of the 67 routable models (slugs joined to source rosters with resolver-style normalization: creator-prefix strip + mode/effort/date suffix handling + exact significant-token match):

The dark 32 is structurally the current open-weight generation, absent from every source (AA/LiveBench haven't cycled the open mid-tier; OpenLLM froze before these shipped):

Why popular models look "unbenchmarked": they are benchmarked, but (1) self-reported by creators in model cards/blogs (non-uniform, self-interested, unstructured: we refuse as authoritative); (2) covered by uniform harnesses for a DIFFERENT size/version than the exact variant we serve; or (3) genuinely uncovered (ByteDance, brand-new drops, the current open mid-tier). Only a minority of the apparent gap was a naming join failure (a few Qwen3 sizes), now recovered; the aggregate barely moved (33 -> 35), so the gap is real.

Ingestion: the NORMAL pipeline, no new mechanism (corrected 2026-05-29). An earlier draft here overstated this as a bespoke "ingestion mechanism." It is not. LiveBench and OpenLLM are just two more benchmark-only sources flowing through the existing IngestService, exactly as AA does today. ingest_service.py already routes any benchmark_-prefixed field to model_benchmarks keyed by a per-source suite (suite = source_type minus _api), attached to the model_base the IdentityResolver matched. So each new source becomes its own suite (livebench, open_llm_leaderboard) and associates with the matching base as another benchmark. Standard onboarding per source: (1) a source_registry row, role enricher, listed in BENCHMARKS_ONLY_SOURCES; (2) a formatter emitting benchmark_<source>_ fields plus a creator_hint/model id the resolver can match.

The "don't pollute the catalog" guarantee is AUTOMATIC from the existing design, not something we build:

• Enrichers/benchmarks-only sources NEVER bootstrap a base. An unmatched row is DROPPED, not created, so a mis-named row is a miss (lost coverage), never a wrong base or phantom model.
• is_benchmarks_only skips field_data, so these sources cannot touch the spec the materializer reads. Scores live in their own table/suite, never golden tables.

The identity matching is the resolver's existing job (creator + model_key + variant_detector), far more robust than the throwaway regex used for the coverage estimate above; the false positives I saw were artifacts of that regex, not the real resolver. For ambiguous names the lever is the formatter supplying a better creator_hint/id (the same tuning AA's _extract_model_suffix does); worst case is a safe dropped row.

The ONE genuinely new piece is DOWNSTREAM of ingestion, in E3 not the pipeline: catalog_evidence.py currently reads only AA's metrics. To consume LiveBench/OpenLLM it must read across MULTIPLE suites and map each suite's columns to task families (e.g. LiveBench's summarize column -> Summarization). That projection change is the only real work beyond a per-source formatter.

Net: AA + LiveBench + OpenLLM realistically reaches ~half the routable fleet (frontier + last-gen open). The current open generation stays dark until self-run lm-eval-harness on our own routes (post-launch), which is both the only path to that half and the differentiation wedge: uniform, trustworthy, exact-served-variant benchmarks for production open models.

Dependency-gated follow-ups (noted 2026-05-29)

Tasks surfaced by this session that are deferred or depend on something else completing first. Distinct from the V2 backlog (I7) below; this is the concrete near-/post-launch dependency chain for the engine + benchmark coverage.

Existing V2 items (active perf probes, custom/fine-tuned classifier, E8 full feedback loop, HELM-style Closed QA / Classification eval design) remain as listed under I7 / the parked research thread; not duplicated here.

AA ingest is broken, and fixing it beats adding sources (2026-05-29) — RESUME HERE

Decisive finding that supersedes most of the multi-source plan above. AA's LIVE leaderboard already covers the current open generation: 370 models / 227 open-weight, confirmed to include Qwen3/3.5/3.6, Gemma 3/4, Llama 4 Scout/Maverick, Doubao (ByteDance) Seed, Mistral Small, Nemotron. The dark fleet is NOT unbenchmarked; our AA INGEST is dropping it.

Diagnosis (prod, model_benchmarks suite=artificial_analysis): only 82 rows, all from a single one-shot ingest on 2026-05-20 (not refreshing). Coverage by creator: openai 31, deepseek 21, google 10, minimax 6, nvidia 6, anthropic 5, mistral 2, cohere 1. ZERO rows for qwen, meta, bytedance, moonshot, despite all being approved creators with many routable models.

Root cause = creator-alias bug in app/formatters/base.py CREATOR_SLUG_BASE (used by the AA formatter). AA labels creators by parent company; our catalog uses brand slugs:

• "alibaba": "alibaba" is orphaned. AA calls Qwen "alibaba"; we call it "qwen". Every Qwen AA row resolves to a non-existent "alibaba" creator and, since AA is an enricher that never bootstraps, is DROPPED. This is the entire ~17-model Qwen dark set.
• moonshot/kimi absent from the map → Kimi dropped.
• bytedance mapped but AA likely uses "doubao" → dropped.
• meta IS mapped (meta-llama -> meta) yet still 0 rows -> a DIFFERENT failure (model_key matching, not creator alias); needs separate diagnosis.

The fix (no new source needed): (1) correct the alias map: alibaba -> qwen, add moonshot/moonshotai/kimi -> moonshot, doubao -> bytedance; (2) diagnose Meta's model_key matching; (3) re-run AA ingest from /admin/ingestion. Expected to lift routable AA coverage from 14 toward ~50-60 of 67, with zero new source, one uniform source, riding the existing pipeline, auto-refreshable. This validates the "validate before building" instinct: we nearly built LiveBench+OpenLLM ingestion to fill a gap that is mostly a one-line alias bug.

Second AA fix (atomic-set refresh): atomic metrics are NOT available for all models even within AA (our 82 rows: gpqa/hle/scicode ~74, mmlu_pro 52, aime 32). Two-tier consequence: the composite intelligence_index is present for ~all models -> always gives the overall generalist signal (servable, LOW, no 503); per-family real signal is gated by atomic availability. This CONFIRMS keeping intelligence_index as the overall axis (don't drop it; it's the floor for atomic-less models) and that confidence tiers naturally encode atomic availability. SEPARATELY: our formatter extracts only the OLD atomic set (mmlu_pro, gpqa, hle, livecodebench, scicode, aime, math_500). AA v4 added atomics we don't capture, two of which fill families we'd written off: IFBench -> instruction_following, AA-LCR (long-context) -> reading_comprehension -> CLOSED QA, Terminal-Bench -> agentic code. Extracting AA's current atomic set could fill more families from AA alone, further shrinking any need for other sources.

RESUME-HERE next step (unstarted): verify against ONE live AA API record (we have ARTIFICIAL_ANALYSIS_API_KEY) both (a) the creator slugs (alibaba/doubao/moonshot?) and (b) what atomic benchmark fields the API returns today (does it expose IFBench/AA-LCR/ Terminal-Bench per model?). That single pull tells us exactly what the two fixes recover before touching code. Then: fix CREATOR_SLUG_BASE + formatter atomic extraction, diagnose Meta, re-run AA ingest, re-measure routable coverage.

Revised source strategy: AA-ingest-fix is now the primary coverage lever (cheapest, most maintainable, one uniform source). LiveBench/OpenLLM demoted to "only if AA-full still leaves gaps," additive-precedence not blend. Self-run lm-eval-harness remains the durable answer for our exact served variants + anything AA misses.

E3 mapping + confidence model LOCKED (2026-05-31) — supersedes the provisional Part 1 above

This locks the E3 Part 1 mapping and, separately, replaces the count-based confidence tier with a continuous evidence weight. Both are implemented in app/services/router/{catalog_evidence,eligibility,scoring,taxonomy}.py (42+ router unit tests green; validated end to end over prod model_benchmarks).

Locked mapping (family -> skill -> atomic metric). Three layers. A skill is the mean of its rank-normalized atomics; a family is the mean of its skills.

Composites coding_index / math_index are dropped (double-count their own atomics); aime / math_500 / math_index are parked (NVIDIA has no math family). The 9 proxy families route on intelligence_index rather than staying silent, because a silent family hard-fails to HTTP 503 on very common prompts.

Why count-based confidence tiers were killed. Measured on prod (82 AA rows), the effective independent metric count per skill (Kish formula on the real correlation matrix) tops out at 1.30:

So the whole LOW/MEDIUM/HIGH ladder spans ~1.1 to 1.2 real observations. Two consequences: (1) a "HIGH = 3 independent metrics" tier is physically unreachable with AA data; (2) re-adding coding_index to reach HIGH (the old "Path B") buys 1.14 -> 1.18 effective N for a MEDIUM -> HIGH tier jump, i.e. it games the evidence model. Path B is rejected; metrics stay honest (Path A).

The replacement: evidence-weighted single gate. The two-gate model (floor on score AND a confidence-tier veto) is collapsed into one comparison. Confidence becomes a small continuous discount, not a hard veto, which also fixes the frontier-model bias (a high-scoring single-atomic cell like GPT-5.5 code was rejected by the old veto despite being the best coder):

eff_n           = k / (1 + (k-1) * mean_pairwise_r)   # de-correlated metric count, >= 1
evidence_weight = 1 - PENALTY[preset] * (1 / eff_n)   # in (1-penalty, 1]
effective_score = raw_score * evidence_weight
eligible        iff effective_score >= FLOOR[preset]   # single gate, no tier veto

Penalties are small by necessity (eff_n's range is narrow and the floors are high; a large penalty puts the floor out of reach). The penalty is preset-graded so it operationalizes the enterprise/startup split on the knob we already have: Strict discounts thin evidence hardest (predictability), Permissive barely (outcome quality). Floors and penalties stay internal and tunable; the pipeline shape is the contract. F2 relaxation is unchanged (it now lowers floor + penalty together). ConfidenceTier is removed; the quality cell carries eff_n (float)

• metric_count (audit only). E5 scoring's quality tier uses effective_score too, so better-measured cells get their slight edge in ranking.

End-to-end on prod: GPT-5.5 code (raw 0.97, eff_n 1.0, single atomic) clears all presets including Strict (0.876); old DeepSeek V3 code (31st percentile) fails correctly; Summarization is now servable. 860 cells project (vs the handful when 9 families were silent).

Deferred (V2, with explicit triggers). Benchmark recency/decay (moot today, single 2026-05-20 snapshot); per-metric benchmark-quality priors; score- consistency term (median within-skill spread is ~0.04, near-zero signal). When self-run evals add genuinely independent benchmarks, eff_n can rise past 1.3 and the weight differentiates more; no code change needed.

Platform north-star vs V1: triage of the 10-point "intelligence platform" analysis (2026-05-31)

A strategic analysis argued the leap from router to benchmark-intelligence platform is the real moat (evidence quality, uncertainty, production feedback, governance), not more routing logic. Agreed as the V2 direction. Triaged against the locked roadmap, most items are already V1-core, already deferred for dependency reasons, or contradict a locked decision:

Reality check captured for the resume note: the highest-ROI work right now is still finishing the engine so it runs end to end (wire the E1 classifier model + the assembly orchestrator + chain execution), which precedes every platform item above. Routing-confidence (#4) is the one new V1 task this analysis adds beyond tasks 1-2 here.

E4: Latency data source (LOCKED 2026-05-28)

What E4 is solving

Engine needs per-(model × provider) latency data so:

• mode=latency (D14) can rank routes by speed
• mode=balanced (E5) can weigh latency alongside quality and cost
• Shadow reports (D9) can substantiate latency claims for a switching opportunity

Two metrics matter:

Existing infrastructure (not greenfield)

Catalog already has the relevant pieces:

So the passive-observation path exists. The gap is the cold start: launch day 1, new router has no customer traffic, no observed samples.

Data source at launch: AA published + passive logs overlay

   AA published values (TTFT_p50, TPS_p50 per route)
        │
        │ at launch: this is all we have
        ▼
   route.latency = AA published
        │
        │ as inference_usage_logs accumulates customer traffic
        ▼
   route.latency = observed p50 from logs       (when samples ≥ 5 in 24h)
                 fallback: AA published         (when samples < 5)
                 fallback: unknown / sorted last (when neither)

AA already provides TTFT and tokens_per_second per (model, provider) alongside the quality metrics E3 ingests. Same vendor relationship, same ingest pipeline extended to capture latency columns.

The fallback chain (observed → AA → unknown) is the honest hierarchy: real observation on this route is the truest signal; AA's published value is a reasonable cold-start proxy; absence sorts the route last for latency-sensitive modes.

Storage shape: per-route, not per-model

Latency varies by provider for the same model (Llama-3.3 on Together ≠ Llama-3.3 on DeepInfra). Storage MUST be keyed on (model_id, vendor) (i.e., the route), not model alone. Implementation detail (new columns on model_routes vs sister table TBD at build time); the design constraint is the granularity.

Each row carries provenance: whether the value is AA-derived or observed-derived, and the timestamp. This lets the engine prefer fresher / higher-confidence data and lets the dashboard explain the source.

Freshness

No new scheduling. Reuse what already exists. Latency drifts faster than quality (provider congestion, hardware updates), which is exactly why the observed overlay matters more here than in E3: once we have traffic, our observations are more current than AA's last refresh.

Active perf probes: deferred (not at launch)

Synthetic perf probes (we send periodic canary requests purely to measure latency on low-traffic routes) are deferred. Reasons:

• Cost: ~$36/day at hourly probes across 50 models × 5 providers × $0.001/probe
• AA + passive overlay covers the cold-start and steady-state cases
• Probe data is best-case (canary, not customer-shaped) which gives a misleading floor

Future-enhancement triggers — when active perf probes start mattering:

The probe infrastructure exists (backend_health_check); extending it from liveness to perf measurement is wiring, not new architecture.

Shadow report substantiation for latency claims

Substantiation rule for mode=latency switch opportunities in the report:

Matches D6 honesty contract. We claim what we can substantiate; we omit what we can't.

Response surface (cross-check with D15)

Routing-decision rationale ("picked M because lowest TTFT among substantiated candidates") lives in the routing-decisions API (GET /v1/routing-decisions/{id}) and the dashboard. The OpenAI-compatible response body stays minimal per D15. Latency metadata does not bloat per-call responses.

Sub-decisions locked

E5: Balanced-mode scoring function (LOCKED 2026-05-28)

What E5 is solving

After E2 filters the eligibility pool (preset floor + capability + health + substantiation), and quality (E3) / cost (model_pricing) / latency (E4) are all available per candidate, the engine needs to pick one. The cost, quality, and latency modes are single-axis sorts and trivial. Balanced is the one that needs structure, and it's the default mode per D14 — likely ~80% of routed traffic.

Decision: multi-stage lexicographic filter, not weighted sum

   mode=balanced pipeline
   ──────────────────────

   Eligibility pool (preset E2 floor + D6 substantiation + capability + health)
        │
        ▼
   1. LATENCY OUTLIER FILTER
      Drop candidates with TTFT > 3x pool median
      (rejects routes having a bad day)
        │
        ▼
   2. QUALITY TIER FILTER
      Keep candidates with quality_score(M) ≥ 0.9 × max(quality_score) in pool
      (top 10% quality tier of what passed the floor)
        │
        ▼
   3. COST MINIMIZATION
      Sort remaining ascending by total token cost
      cost = input_tokens × input_price + output_tokens × output_price
        │
        ▼
   4. LATENCY TIEBREAKER
      Within 10% of cheapest, prefer lower TTFT
        │
        ▼
   Pick top

Plain-English version (goes in customer docs):

Balanced mode finds the cheapest candidate that's within 10% of the highest-quality option in your eligible pool. Among similarly cheap candidates, the faster one wins.

Alternatives considered and rejected

The multi-stage filter is intelligible at every step: customer reads the dashboard audit and sees "step 1 kept 8 of 12 candidates; step 2 kept 3 of those by quality tier; step 3 picked the cheapest of those 3; latency tiebreaker did not change the pick."

Other modes share the same pipeline

All modes share the eligibility pool from E2; modes differ only in post-processing. One pipeline, mode-specific parameters.

Why these specific numbers (launch placeholders)

Tunable post-launch based on Path 3 feedback. Same posture as E2's floor values.

Per-preset modulation: floor only at launch

The preset (Strict/Standard/Permissive) controls the eligibility floor via E2. Inside balanced mode, the within-pool parameters (0.9, 3x, 10%) are FIXED at launch — they do not vary by preset.

Reason: preset already restricts the pool. Varying within-pool parameters too creates 12 combinations to reason about (4 modes × 3 presets) and complicates the audit trail. Simpler to ship fixed.

If post-launch evidence shows Strict customers want a tighter quality tier (e.g., 0.95) and Permissive want looser (0.80), preset-modulated parameters can be added as a follow-up. Not launch work.

Transparency: publish the structure, keep parameters mutable

Matches D17's "product transparency, not business transparency" principle. Customers see the logic; they don't need to see the parameters change underneath them.

Sub-decisions locked

E6: Fallback chain construction (LOCKED 2026-05-28)

What E6 is solving

When the primary pick fails (5xx, 429, timeout, route went unhealthy mid-flight), the engine needs to try another route. D16 already locks the streaming half (silent fallback pre-content, hard fail post-content). E6 fills in the rest: how the chain is built, how long it is, which errors trigger fallback, what happens when the chain is exhausted.

Decision: static top-3 chain, hard fail on exhaustion

   E6 fallback flow
   ────────────────

   Routing decision time:
        run E5 pipeline once, take top 3 candidates ordered
        primary  = #1
        chain    = [#2, #3]
        store full chain in routing-decision record (audit)

   Attempt loop (per request):
        for route in [primary, #2, #3]:
            if route now unhealthy (E2 health re-check):
                skip
            try route with per-attempt timeout
            if success:
                return response
            if error class is "do-not-fallback" (400/401/403/moderation):
                propagate to caller, stop
            else:
                continue to next route
        # chain exhausted
        return HTTP 503 + structured error

   Customer's SDK (OpenAI, Anthropic, LangChain, ...) handles 503 via
   built-in retry with backoff. Retry hits our pipeline as a fresh request.

Alternatives considered and rejected

Error classes

Fallback triggered:

Fallback NOT triggered, propagate to caller:

Streaming interaction (already locked at D16)

Pinned-route chain

When the request uses a literal model pin (model="gpt-4o", not auto), the chain is "same model, different healthy provider" computed via Path 1 (provider arbitrage). Never cross-model. Customer pinned for a reason; pin is honored.

For auto and auto:<mode>, the chain is the top-3 from E5's pipeline as described.

Timeouts and deadline budget

Decreasing per-attempt timeouts leave room for chain exhaustion within budget. Numbers are launch placeholders; tunable post-launch based on observed timeout distributions.

Circuit breaker — handled at E2 eligibility, not E6

Routes that fail repeatedly trip a circuit breaker and are excluded from the eligibility pool by E2's health check before E5 even sees them. E6 inherits the already-filtered pool and does not re-implement circuit-breaker logic. Single source of truth for "is this route trustworthy right now."

"App handles retry" — what 503 actually means for the caller

The customer doesn't write retry logic in most cases — the SDK does it:

from openai import OpenAI

client = OpenAI(
    base_url="https://inferbase.ai/v1",
    api_key="ibk_...",
    max_retries=3,        # built-in retry; default is 2
)

# Will retry automatically on 503 / 429 / timeout
client.chat.completions.create(model="auto", messages=[...])

Same pattern for Anthropic SDK, LangChain, LlamaIndex, every major client. Each retry is a NEW request through our pipeline (re-classify, re-route, new chain). By the time the retry lands:

Known nuance: a mid-response failure (we got partial bytes, then connection dropped) may mean upstream processed and billed tokens without us delivering them. Customer retries and gets billed twice. Industry-standard problem; idempotency keys are a phase-2 fix, not launch work.

Sub-decisions locked

E7: Routing decision caching (LOCKED 2026-05-28)

What E7 is solving

Same prompt arrives twice. Do we cache the routing decision (the chain of 3 from E5/E6) and reuse it, or re-run the full pipeline each time?

Validate-before-building: what does the pipeline actually cost?

With E1's classifier cache (24h TTL, content-addressed) warm:

Against 200-2000ms downstream TTFT, 10ms is invisible and 100ms is acceptable. The expensive stages are already cached at the data layer.

Decision: no decision cache at launch

The subtle correctness issue with caching routing decisions:

   With a 5-min decision cache keyed on prompt + customer
   ──────────────────────────────────────────────────────

   T=0:00   Prompt P → pipeline → routes to Llama on Together
            decision cached for 5min

   T=2:00   Together has a hiccup; E4's overlay shows TTFT tripled

   T=3:00   Same prompt P → cache HIT → still routes to Llama on Together
            despite the data layer knowing it's degraded right now

   Pipeline running fresh at T=3 would have picked DeepInfra.
   The cache froze the decision against fresh evidence.

The data-layer caches (E1 24h classifier, E4 5min latency, E3 quarterly quality) are appropriate because their INPUTS change at those cadences. The DECISION should always reflect current inputs. Layering a decision cache on top means staleness compounds.

Alternatives considered and rejected

What stays cached at launch (already locked at other decisions)

These compose to "pipeline reads cheap, decision is computed fresh, answer reflects current state." No additional cache layer needed.

Auto-routing variance: same prompt may route to different models

A real consequence of no decision cache + observation-driven engine: identical prompts arriving at different times may route to different models. This is the industry-standard behavior of every auto-router (OpenRouter, NotDiamond, AWS Bedrock IPR). It is the honest tradeoff for routing on VALUE rather than CONSISTENCY.

Customer mitigations available in our design:

Customer-facing docs explicitly call out this variance so it isn't a launch surprise. Shadow replay (D8) lets the customer SEE the variance before they go live ("we would have routed to Llama 73%, Qwen 21%, GPT-4o-mini 6% across your 5,000 prompts; 47 of 50 sampled-validated outputs were equivalent quality across these picks"). If the variance is acceptable on their data, they flip live; if not, they pin.

Plain-English version for customer docs

Each request runs through the routing pipeline fresh. Inferbase caches the underlying data (model competence, observed latency, classifier output) at appropriate cadences, so the pipeline is fast — typically ~10ms before your prompt reaches the model. We deliberately do NOT cache the routing decision itself; that would mean serving you a stale pick when route conditions have changed.

A natural consequence: the same prompt sent twice may route to different models if route conditions changed between calls. This matches the behavior of every auto-router in the market. If you want consistent routing for a specific endpoint, pin to a specific model (model="llama-3.3-70b") — Inferbase still picks the cheapest healthy provider for that model on every call.

Sub-decisions locked

E8: Path 3 evidence integration (LOCKED 2026-05-28)

What E8 is solving (and not)

Path 3 (D7) sample-call validations produce customer-specific evidence: "for customer C on summarization, M was equivalent to Z on 47 of 50 of THEIR prompts." Original formulation of E8 was: how does this evidence feed back into live routing decisions to make Path 2's catalog signal customer-tuned over time?

After validate-before-building: the launch lock SEPARATES the two uses of Path 3 evidence and ships only the first:

Why the lighter shape

Path 3 evidence is generated regardless of whether routing consumes it. Customers see the verification in their shadow report (D9 evidence tier display); they decide whether to flip live based on that evidence. Live routing then uses E3's catalog signal — which is itself validated by shadow's Path 3 evidence (M was a candidate because catalog ranked it high; Path 3 confirmed it on customer data; live routing picks the same M for the same reason).

The path from "verify" to "route" goes through customer review, NOT through automated feedback. This is more honest:

   Without E8 routing integration (launch shape):
   ───────────────────────────────────────────────
   Path 3 validates M → report shows evidence → customer reviews
        → customer flips live → live routing picks M
        (because catalog signal that suggested M is the same signal that picks M live)

   With E8 routing integration (deferred):
   ───────────────────────────────────────
   Path 3 validates M → report shows evidence → customer reviews
        → customer flips live → live routing picks M
        (because catalog + customer's path3 blend says M)

The difference at launch is small. The complexity cost of E8 integration is real (per-customer quality_score blends, calcification mitigation, cross-customer aggregation, exploration mechanism). Validate-before-building says: ship the storage and display, defer the routing integration.

Alternatives considered and rejected (for launch)

What ships at launch

Activation triggers (when to revisit full E8)

The full E8 (per-customer blend + cross-customer aggregate + exploration) gets revisited when at least one of the following fires:

Until one of these fires, E8's routing-feedback layer is not earning its complexity.

Sub-decisions locked

What's preserved for future activation

When triggers fire, the full E8 stack (blend at quality_score lookup, cross-customer aggregate suite, epsilon-greedy exploration with new-candidate bootstrap) is in the backlog with the design already worked through. The lighter launch shape doesn't prejudice the architecture; it just defers the engineering until signal warrants.

Engine assembly (orchestrator) — IMPLEMENTED 2026-05-31

The orchestrator (app/services/router/engine.py, assemble_decision) ties the pure E-series stages into one deterministic pipeline. It does no I/O beyond the injected classifier; routes + catalog evidence + latency are passed in (loaded and cached by the request layer), so the whole decision is unit-testable without a database. Output is a RoutingDecision (the unit the execution loop and the M1 audit record consume).

messages ─▶ classify (E1, cached, F1 fallback)
              │  task_family
              ▼
        attach catalog quality cell per route (E3)         routes + evidence ─┘
              │
              ├─ Auto pool ───▶ select_eligible(preset)  (E2 + F2 relaxation)
              │                    │ eligible, preset_used, floor_drops
              └─ Custom/pinned ─▶  skip E2 (pre-vetted)
              ▼
        score(eligible, mode)   (E5: balanced pipeline | cost/quality/latency sorts)
              ▼
        build_chain(ordered, same_model_only)   (E6: primary + 2 fallbacks)
              ▼
        RoutingDecision{ classification, mode, chain, eligible_count,
                         preset_requested, preset_used, floor_drops,
                         routing_confidence }

routing_confidence (analysis item #4) is the decisiveness of the pick: the primary's relative margin over the runner-up on the mode's decision axis (cost margin for cost/balanced, evidence-discounted quality margin for quality, TTFT margin for latency). 1.0 when a single candidate qualified (no contest), None when unservable, near 0 when it was a close call. Computed on the chain (post-pin-filter) so a pin is judged against its actual same-model fallback.

is_servable is bool(chain); an empty chain (F2 exhausted, or a silent family with no quality cell) is the caller's 503 no_eligible_candidates signal. The E5 mode dispatcher (scoring.score) is also completed here: balanced is the structured pipeline, cost/quality/latency are single-axis sorts with deterministic tiebreakers. 55 router unit tests green.

Chain execution loop — IMPLEMENTED 2026-05-31

app/services/router/execution.py drives the E6 chain against real inference backends. Backends are resolved per candidate via the injected backend_for(vendor) (the registry's get_backend_by_vendor), so the loop is testable with fakes. Two entry points share the E6 policy:

for each candidate in chain (in order):
    timeout = attempt_timeout(index, elapsed, total_deadline)   # 15/10/5s within 30s
    if timeout <= 0:                      -> deadline exhausted -> TIMEOUT (504)
    backend = backend_for(vendor)         -> None: record FAILED, fall back
    if health_check and not healthy:      -> SKIPPED_UNHEALTHY, fall back   (per-attempt re-validation)
    try serve:
        success                           -> SERVED (index 0) | FALLBACK_SERVED
    on error:
        should_fallback(status, timed_out) and not last -> fall back
        not should_fallback (4xx/auth/moderation)       -> propagate, HARD_FAIL
        retryable but last route                         -> exhaustion
chain exhausted -> HARD_FAIL (503), or TIMEOUT if the last attempt timed out

Each attempt is recorded as an AttemptRecord{candidate, outcome, status_code, latency_ms, error} with AttemptOutcome in {served, failed, timed_out, skipped_unhealthy} for the M1 trail (F8). Disposition -> HTTP mapping (503 on exhaustion, propagate an upstream 4xx, 504 on deadline) is the API layer's job. 11 execution unit tests (fallback, propagate, timeout, health-skip, the three D16 streaming cases). The per-attempt circuit breaker stays E2's job (E6 lock).

Live request handler + M1 persistence — IMPLEMENTED 2026-05-31 (engine is now non-inert)

The engine is wired to an additive HTTP surface, leaving the prod /inference/chat/completions (old router) untouched until cutover (I5):

POST /api/v1/inference/router/chat/completions   engine-routed completion (stream + non-stream)
GET  /api/v1/inference/routing-decisions/{id}     the M1 decision record (D15)

Flow (app/services/router/serve.py, thin endpoints in app/api/v1/router_inference.py):

auth (require_inference_credits)
  -> load pool candidates from model_routes + cached catalog evidence (loader.py)
  -> assemble_decision(classifier=None -> F1, ...)        [engine]
  -> execute_chain / stream_chain                          [execution, D16]
  -> persist routing_decisions row (M1/F8) + record_usage_and_deduct (credits)
  -> response.id = req-<uuid> (the D15 handle); response.model = "vendor/upstream"

M1 mapping persists classification (F1 status surfaced honestly), mode/preset, eligibility_pool_size, primary route, final_disposition, routing_confidence (new dedicated column, additive migration a7f2e1c9d4b6), latency, cost, the chain + per-attempt trail, and the F2 floor drops. Every request writes a row, success or failure (an empty pool writes a hard_fail row then returns 503). Disposition -> HTTP: 503 exhaustion / propagate upstream 4xx / 504 deadline.

Catalog evidence is projected once and cached in-process (loader.py, 10-min TTL) so the request path never re-projects. 8 new tests (service-layer M1 persistence + HTTP smoke + D15). The handler passes classifier=None (E1 ONNX not wired) so routing currently classifies every prompt as Other (F1) -> the overall proxy; wiring the model is the next task and needs no handler change.

Still downstream: swap the new path onto /inference/chat/completions + delete the old router (cutover, I5); key-scoped preset; ttft_ms on the streaming path.

E1 classifier WIRED 2026-05-31 (real per-task routing)

The NVIDIA prompt-task-and-complexity-classifier now runs in-process, so routing classifies real task families + complexity instead of the F1 default.

• Model: multi-head DeBERTa-v3 emitting our 11 TaskFamily labels (+ an "Unknown" -> OTHER) and a 0-1 complexity. The HF repo ships no importable module / auto_map, so the CustomModel classes are vendored from the model card (app/services/router/nvidia_classifier.py) and loaded via PyTorchModelHubMixin.from_pretrained (which reads the metadata-less safetensors fine, unlike AutoModel). transformers pinned to 4.44.2 (4.46 breaks on that safetensors), CPU-only torch wheel.
• Topology (Railway 8GB/8vCPU confirmed): torch in-process. Loaded in a background startup task (main.py) so startup never blocks; until ready (or if disabled / load fails) _state.e1_classifier is None and routing uses F1. Inference offloaded via asyncio.to_thread; repeats absorbed by the M5 cache. serve.py passes _state.e1_classifier (read at call time) into the engine.
• Config: E1_CLASSIFIER_ENABLED / _MODEL / _REVISION (revision pinned).
• Verified locally: real load + classify is correct (Python->code_generation, factual->open_qa, summarize->summarization, classify->classification, brainstorm->brainstorming). Unit tests fake the forward (no weights in CI).

Operator deploy notes (Railway): set HF_HOME to a path on the 100GB volume so the ~1GB weight download (backbone + classifier) happens once and persists across redeploys (otherwise re-pulled per cold start, degraded to F1 until ready); set HF_HUB_DISABLE_XET=1 (the xet transfer backend can hang). The torch deps add ~1.5GB to the image / CI install (inherent to in-process torch). ONNX quantization + a separate microservice remain deferred optimizations.

Failure modes (LOCKED 2026-05-29)

What can go wrong, what the router does about it, what bubbles to the caller.

Already locked elsewhere

Many failure paths are already specified by Engine and Caller decisions; this section references rather than re-derives them:

The eight failure modes below cover the remaining surface area.

F1: Classifier failure

Trigger: NVIDIA classifier service errors, returns garbled output, or exceeds 200ms latency budget.

Behavior: Fall back to heuristic: task_family = "Other", complexity = 0.5 (median). Engine continues with degraded classification. Increment classifier_failures counter for alerting.

Customer visibility: None (silent degradation). Routing-decision record carries classifier_status = "fallback_heuristic" so dashboard audit shows the degradation.

Why not hard-fail: Classifier is non-critical; engine can still route reasonably without per-task tuning. "Other" + median complexity treats the prompt as middle-of-the-road; routing still picks a competent model from the eligibility pool.

F2: Eligibility pool empty after E2 filtering

Trigger: Preset floor + capability + health + customer rules filter every candidate out.

Behavior: Soft-fail by dropping the preset floor one tier (Strict → Standard, Standard → Permissive). Re-run E2. If still empty, hard-fail with HTTP 503 and structured error no_eligible_candidates. Routing-decision record carries the floor-drop trail.

Customer visibility: Successful response if floor-drop saved it (dashboard shows the drop). HTTP 503 with actionable message if both passes failed: "no candidates satisfy your preset for this prompt; consider widening pool or pinning a model."

Why bounded floor-drop: Strict customers occasionally hit empty pools on hard prompts (e.g., specialized capability required, no Strict-pool candidate qualifies). Floor-drop is graceful bounded widening; silently widening to ANY candidate would violate preset intent.

F3: Deadline budget exhaustion

Trigger: Request hits the 30s wall-clock budget per E6.5 before any chain attempt returns successful first-content.

Behavior: Hard fail with HTTP 504 Gateway Timeout (semantically more accurate than 503: we tried, time ran out).

Customer visibility: 504 with structured error carrying routes attempted, per-attempt failure reasons, chain exhaustion status. Customer SDK handles like any 504 with built-in retry/backoff.

F4: Catalog DB / registry connectivity failure

Trigger: Catalog read fails; registry can't refresh.

Behavior: Tiered degradation.

Customer visibility: Success for cache-warm cases; 503 with structured error for cache-cold. NEVER serve auto-routed requests with an unknown or unverified candidate.

Why a 30min cap on stale-serve: Routing without fresh catalog signal means unsubstantiated decisions, which violates D6's honesty contract. 30min is a balance between availability and honesty.

F5: Customer hard-rule conflict

Trigger: Per-key model whitelist (D13) excludes all eligible candidates, OR per-request override specifies a model not in the per-key whitelist.

Behavior: Hard-fail with HTTP 422 Unprocessable Entity. Structured error explains the conflict precisely: "your key restricts to {gpt-4o, claude-haiku}; this request specified model=llama-3.3-70b which is not in the whitelist."

Precedence rule: Per-request override NEVER silently escalates per-key permissions. Key is the source of truth for what the request CAN ask for. Per-call overrides operate WITHIN that envelope.

Why 422: Request is well-formed but conflicts with system constraints (the correct semantic for this case). 400 implies caller's request shape is malformed; 403 implies auth; 422 is "we understand what you asked, can't allow it."

Why not silently override: Customer set the whitelist for a reason (compliance, spend control, vendor agreements). Silently routing outside would create surprise spend, audit failures, or contract violations.

F6: Path 3 evidence generation failure (during shadow replay)

Trigger: During Path 3 sample validation in offline or online shadow:

• Judge LLM rate-limits, errors, or returns unparseable output
• Sample call to candidate model M fails
• Customer's historical response unavailable or malformed (when we planned to reuse historical rather than re-run Z)

Behavior:

Customer visibility: Path 3 evidence with inconclusive status does NOT substantiate report claims under D6. The opportunity falls back to Path 2 (catalog benchmark) evidence only, with a note that customer-specific validation was attempted but incomplete.

Why no retry on shadow path: Shadow is non-real-time; the right move on budget-bounded validation is honest reporting of what completed, not retry-burn on flaky calls.

F7: Capability flag mismatch

Trigger: Catalog claims model M supports a capability (tools, JSON mode, vision, long context). Actual call returns 400 "tools not supported" or similar.

Behavior:

1. Mark the (route, capability) pair as capability_unverified in route stats
2. Trigger E6 fallback chain (treat as a route-side failure, NOT a caller-side 400)
3. After 3 distinct customers hit the same mismatch on the same route, auto-update catalog's capability flag to false for that route
4. Alert ops

Customer visibility: Successful response from fallback (if chain has a capability-supporting alternative). 503 chain exhaustion if no capable alternative. Customer never sees our catalog error as a 400.

Why not propagate as 400: The customer's request shape is correct (they're asking for tools and tools are a documented capability of the catalog model). The 400 is OUR catalog data error; falling back is honest self-healing, not error masking.

Why auto-correction needs 3 distinct customers: Single-customer mismatch could be a transient provider issue or a specific edge case. Three independent reports indicates systemic catalog drift; auto-flip prevents the same failure from recurring across the fleet.

F8: Audit trail under failure

Structural requirement, not a conditional behavior. Every routing decision — success OR failure — writes a record to routing_decisions with full context:

Customer visibility: All records accessible by request ID via D15's /v1/routing-decisions/{request_id} API and the dashboard. Failed requests are FIRST-CLASS records, not exceptions buried in error logs.

Why this matters: The wedge IS product transparency. Customers debugging "why did my call fail" need full visibility into what the engine tried and why it stopped. This also serves the post-launch tuning loop — every failure recorded is a tuning signal for E2 floors, E5 parameters, and the activation triggers in E8.

Alternatives considered and rejected

Data model (LOCKED 2026-05-29)

What state the router reads, writes, and persists.

Overview

   ┌─────────────── CATALOG (mostly existing, reused) ─────────────────┐
   │  model_base · model_routes · model_pricing · model_benchmarks     │
   │  model_identities · source_registry · creators                    │
   └───────────────────────────┬───────────────────────────────────────┘
                               │ read by router
   ┌───────────────────────────▼───────────────────────────────────────┐
   │                       ROUTING (new + extended)                    │
   │  ┌───────────────┐  ┌──────────────────┐  ┌────────────────────┐  │
   │  │ api_keys (D13)│  │ routing_decisions│  │ classifier_cache   │  │
   │  │               │  │ (D15 + F8)       │  │ (E1, Redis)        │  │
   │  └───────────────┘  └──────────────────┘  └────────────────────┘  │
   │  model_routes extended with ttft_ms_p50, tps_p50,                 │
   │  latency_provenance, latency_updated_at (E4)                      │
   └───────────────────────────────────────────────────────────────────┘

   ┌─────────────── EVIDENCE (customer-specific) ──────────────────────┐
   │  customer_path3_evidence (E8) · customer_prompt_samples           │
   │  (raw content; 30d auto-purge per D11)                            │
   └───────────────────────────────────────────────────────────────────┘

   ┌─────────────── OPERATIONAL (existing + new) ──────────────────────┐
   │  inference_usage_logs (E4 passive) · backend_health_check         │
   │  benchmark_contamination_denylist (E3.5) · capability_mismatches  │
   │  (F7)                                                             │
   └───────────────────────────────────────────────────────────────────┘

Naming and taxonomy commitments (apply to code AND schema)

Vishal's lock 2026-05-29: identifiers must be unambiguous and self-describing. Don't reuse a word for unrelated concepts. Lock vocabulary at design time. The following commitments apply across the router subsystem:

Single-word concepts that EARN their distinctness through domain specificity stay: key_type (api_keys ∈ inference/analytics/admin), verdict_state (Path 3 conclusive vs inconclusive), classifier_status (ok / fallback_heuristic per F1).

M1: routing_decisions schema

One row per request. Structured columns for common fields; JSONB for variable trails.

routing_decisions
  id                       UUID PK
  request_id               TEXT (the public/external request id, indexed)
  customer_id              UUID FK
  api_key_id               UUID FK
  routing_mode             TEXT (cost / quality / balanced / latency)
  preset                   TEXT (strict / standard / permissive)
  classifier_task_family   TEXT
  classifier_complexity    NUMERIC
  classifier_status        TEXT (ok / fallback_heuristic per F1)
  eligibility_pool_size    INT
  primary_route_id         UUID FK model_routes
  final_disposition        TEXT (served / fallback_served / hard_fail / timeout)
  latency_ms               INT (total wall-clock; nullable on hard_fail)
  ttft_ms                  INT (nullable)
  cost_usd                 NUMERIC (nullable on hard_fail)
  filter_trail             JSONB (each E2 step's input + output sizes)
  chain_attempted          JSONB (list of route ids in order)
  attempt_outcomes         JSONB (per-attempt: route_id, result, latency, error_class)
  customer_rules_applied   JSONB (whitelist hits, per-call override resolution)
  path3_evidence_consulted JSONB (per E8 launch shape: display only; not blended in)
  f2_floor_drops           JSONB (nullable; preset → preset trail)
  f4_catalog_staleness     JSONB (nullable; staleness at decision time)
  f7_capability_flags      JSONB (nullable; capability_unverified marks)
  created_at               TIMESTAMP

Indexes:
  PK id
  UNIQUE request_id
  (customer_id, created_at DESC)
  (customer_id, api_key_id, created_at DESC)

M2: customer_path3_evidence schema

customer_path3_evidence
  id                          UUID PK
  customer_id                 UUID FK
  candidate_model_id          UUID FK model_base
  baseline_model_id           UUID FK model_base
  task_family                 TEXT (NVIDIA's 11 categories per E1)
  equivalence_rate            NUMERIC (e.g., 0.94)
  sample_count                INT
  candidate_preferred_count   INT
  baseline_preferred_count    INT
  verdict_state               TEXT (conclusive / inconclusive per F6)
  last_run_at                 TIMESTAMP
  created_at                  TIMESTAMP

Indexes:
  PK id
  UNIQUE (customer_id, candidate_model_id, baseline_model_id, task_family)

M3: Latency on model_routes (E4 extension)

Extend the existing model_routes table:

model_routes
  ... existing columns ...
  ttft_ms_p50              INT (nullable)
  tps_p50                  INT (nullable)
  latency_provenance       TEXT (aa_published / observed_24h / unknown)
  latency_updated_at       TIMESTAMP (nullable)

The 24h time-series of observed latency lives in inference_usage_logs (existing). Routes carry only the decision-time value the engine consumes per E4.2.

M4: api_keys table (D13)

api_keys
  id                       UUID PK
  customer_id              UUID FK
  key_hash                 TEXT (SHA256 of the raw key; never store raw)
  prefix_display           TEXT (e.g., "ibk_a1b2..." — first 4 chars after prefix)
  key_type                 TEXT (inference / analytics / admin)
  name                     TEXT (customer-set, default "default")
  scopes                   JSONB (D13's rich scoping: env_tag, model_whitelist,
                                   rate_limit_rpm, spend_cap_monthly_usd,
                                   per_route_permissions)
  spend_cap_monthly_usd    NUMERIC (nullable; denormalized for fast checks)
  rate_limit_rpm           INT (nullable; denormalized for fast checks)
  is_active                BOOLEAN
  created_at               TIMESTAMP
  last_used_at             TIMESTAMP (nullable)
  revoked_at               TIMESTAMP (nullable)

Indexes:
  PK id
  UNIQUE key_hash
  (customer_id, is_active)

Hashing matches industry standard (Stripe, GitHub, OpenAI). Rotation = revoke + issue new; we never need to decrypt a stored key.

M5: Classifier cache (E1)

Redis-backed (logisync-redis, already in stack). Not a Postgres table.

Key:    SHA256(prompt + system + tools_schema_hash)
Value:  JSON { task_family: TEXT, complexity: NUMERIC, classified_at: TIMESTAMP }
TTL:    86400 seconds (24h per E1 lock)

M6: Shadow prompt storage (raw content)

customer_prompt_samples
  id                         UUID PK
  customer_id                UUID FK
  prompt                     TEXT
  response                   TEXT (nullable; customer's historical response if
                                    available, else null and we re-run Z)
  classified_task_family     TEXT
  classified_complexity      NUMERIC
  capture_method             TEXT (offline_upload / online_observation)
  upload_batch_id            UUID (nullable; links to offline upload batch)
  created_at                 TIMESTAMP
  purge_at                   TIMESTAMP (= created_at + 30d per D11)

Indexes:
  PK id
  (customer_id, capture_method, created_at)
  purge_at  (for retention sweep)

Raw content stays SEPARATE from routing_decisions because retention class differs (30d for raw, account-lifetime + 90d for decisions per D11).

M7: Contamination denylist + capability mismatches

benchmark_contamination_denylist     (empty at launch per E3.5)
  id                       UUID PK
  model_base_id            UUID FK
  benchmark_metric         TEXT (e.g., "mmlu_pro")
  reason                   TEXT
  added_at                 TIMESTAMP
  added_by                 TEXT

capability_mismatches                (populated by F7 triggers)
  id                       UUID PK
  route_id                 UUID FK model_routes
  capability               TEXT (e.g., "tools", "json_mode", "vision")
  customer_id              UUID FK
  request_id               TEXT
  observed_at              TIMESTAMP
  auto_corrected_at        TIMESTAMP (nullable; set when catalog flag flipped)

Indexes:
  capability_mismatches: UNIQUE (route_id, capability, customer_id)
                         — so "3 distinct customers" trigger is COUNT(*) on it

M8: Retention orchestration

Single daily cron job, three sweeps, aligned to D11 explicit retention windows:

Cron-based rather than per-row TTLs because the policies are class-level, not row-level; one job, one place to reason about retention.

Retention map (D11 applied)

Alternatives considered and rejected

Sub-decisions locked

Observability (LOCKED 2026-05-29)

What's recorded and surfaced across decisions — distinct from F8's per-decision audit trail. F8 is the RECORD layer (one row per request); Observability is the AGGREGATE + DASHBOARD + ALERT layer that derives signal from those records.

Context: shadow vs live (amended at this lock)

This lock incorporates a reframing of shadow replay's role raised 2026-05-29 (see amendments to D8, D10, P10 in the decision log). Practical effect on Observability:

• Shadow is a lead-qualification tool used PRE-conversion. Customer goes through it once (or repeats on demand). Output is the D9 report. There is no ongoing customer-facing shadow dashboard.
• Live is the post-conversion product. Customer-facing dashboard surfaces ongoing routing visibility, usage, costs, per-key analytics.
• Sales-led conversion is the primary path for D1's infra buyer. Self-serve flip-live remains for smaller / dev-first customers.

O1: Instrumentation stack

Sentry (existing inferbase-hy org) for errors and exceptions. Metrics derived from routing_decisions (M1) and inference_usage_logs aggregate queries against Postgres indexes. No separate metrics database at launch. OpenTelemetry distributed tracing NOT at launch.

O2: Customer LIVE dashboard

Per-customer view, accessed by authenticated customer. Surfaces:

O3: Shadow output (report archive, not dashboard)

No live status dashboard. No Path 3 accumulation surface for customers. The report itself carries all the value; everything else is plumbing visibility that doesn't earn the engineering.

O4: Internal ops dashboard

Internal-only, accessed by Inferbase engineering / ops. Surfaces:

O5: SLOs at launch

99.5% is honest for launch; 99.9% requires mature ops + redundancy + incident response. 150ms p95 overhead is invisible against upstream LLM TTFT per E1's accuracy-first stance. Weekly AA ingest is a manageable cadence.

O6: Alerts at launch

Three severity tiers with strict criteria; alert noise is the dominant failure mode of early observability.

O7: Admin notification + sales workflow

New decision area introduced at this lock. Triggered when a customer generates a shadow report.

Alternatives considered and rejected

Sub-decisions locked

Implementation plan (LOCKED 2026-05-29)

How the design above gets built, in what order, with what validation, and what explicitly is NOT in V1.

Context: clean cut, not strangler-fig

Vishal's call 2026-05-29: no paying customers + beta status means we can build new in place and decommission old code in the same cutover. The strangler-fig + per-customer flip mechanism in the original draft was over-engineered for our actual customer state. Internal dogfood is the validation; no customer-protection gate is needed.

Frontend reuse is non-negotiable: the existing portal works; the backend adapts. UI changes are limited to genuinely new surfaces (shadow report viewer, routing-decision detail page).

I1: Build strategy

Clean replace, not strangler-fig. New code goes in at existing route paths; old code is deleted in the cutover PR. Internal dogfood before cutover; shadow comparison against the old code is optional confidence-building, not a customer-protection gate.

I2: Build order in phases

   Phase 0  ─── Foundations (backend)
              M1-M8 migrations, naming constants, D3 adapter interface
                   │
                   ▼
   Phase 1  ─── Engine core (backend)
              E1 classifier → E2 eligibility → E5 pipeline → E6 chain → Path 1
                   │
                   ▼
   Phase 2  ─── Caller surface (backend, existing API routes)
              D12 endpoint, D13 keys, D14 overrides, D15 response + decisions API,
              D16 streaming, F1-F7 error handling, F8 audit trail
                   │
                   ▼
   Phase 3  ─── Shadow replay (backend + minimal new frontend)
              D8 offline + online, Path 2 + Path 3 (LLM-judge), D9 report,
              O7 admin notification (reuses app/admin/leads/)
                   │
                   ▼
   Phase 4  ─── Observability (backend + frontend wiring)
              O2 wires existing app/dashboard/, O3 new report archive page,
              O4 internal ops dashboard, O6 alerting, O5 SLO tracking
                   │
                   ▼
   Phase 5  ─── Hardening + internal beta
                   │
                   ▼
              ~2-week full internal dogfood
                   │
                   ▼
              cutover PR: new code live + old code deleted in same commit set
                   │
                   ▼
              external beta opens

I3: Validation gates (simplified)

I4: Migration mechanics

Removed. No per-customer flip. Direct cutover.

I5: Old-router removal

Part of the cutover PR, not a post-migration step. Delete: routing.py, classifier.py, sanity.py, inference_routing_logs, optimize_mode plumbing, the LLM-classifier-per-call code from [[project_routing_latency]]. Beta-stage license to move fast; no 30d retention period for old code.

I6: Team allocation and target dates

Team: Vishal + 1-2 engineers (current size).

Estimates revisit after Phase 0 completion (foundations always reveal true velocity).

I7: Frontend reuse principle

DO NOT change UI / frontend. The existing portal already works. Backend adapts to existing frontend expectations; minor frontend data-binding updates only.

Reusable as-is:

Genuinely new frontend surfaces (small list):

V1 vs V2 bifurcation

Comprehensive split of what cutover delivers vs what's deferred to post-launch activation triggers.

V1 launch scope (delivered at cutover)

V2 deferred features (activation triggers noted where defined)

What this V1/V2 split does NOT compromise

The wedge content (verified on your data, BYO inference, transparency) is FULLY in V1. What's deferred is mostly:

• Scaling infrastructure (Prometheus, OTel, 99.9% SLO, distributed tracing)
• Activation triggers we don't yet have signal for (E8 routing feedback, active probes, custom classifier)
• Geographic / enterprise expansion (EU, IAM/SSO, SOC 2 Type 2)
• Capability-expansion features (Anthropic endpoint, response cache, session stickiness, fine-tuning services)

None of these are wedge-critical. V1 ships the wedge complete; V2 grows it.

Alternatives considered and rejected (revised list)

Sub-decisions locked

Decision log

Open sub-decisions still pending:

• Shadow-replay design (Q1-Q6). Active discussion as of 2026-05-28. Six product questions inside shadow replay (offline/online, predict/call, report shape, quality prediction, flip threshold, data retention). See Caller and surface section above.

Discarded approaches

Recorded so we do not relitigate them.

• The 4-step seam-based design from earlier 2026-05-28 (FeatureExtractor / QualityEstimator / ProviderRanker / Executor with locked sub-decisions on min_quality, alpha, cooldown durations, error taxonomy, streaming commit, deadline budget, trace schema). Discarded because the seams and knobs were shaped by current code rather than by a fresh derivation from the router's purpose.
• The earlier same-day "enterprise-grade layered architecture" sketch (ingress, L1 feature extraction, L2 policy engine, L3 scoring, L4 execute and failover, L5 post-hoc eval). Same defect: shaped by existing seeds in the repo, not by first principles.