One API for All LLM Providers: What Silently Breaks

For the first few years of the LLM era, choosing a provider meant learning its SDK, its request format, its error codes, and its billing quirks, and then building your application around all four. Committing to a second provider meant doing it twice, plus an abstraction layer to hide the differences from the rest of your code. Teams did it anyway, because no single vendor held the best model for every task for long, and the churn has only accelerated since.

What changed is that the industry quietly converged on a wire format. The OpenAI Chat Completions API, designed for one company's models, has become the closest thing LLM inference has to a standard plug: a single API for multiple LLM providers is now mostly a question of which base URL you point the same SDK at. That convergence is real and genuinely useful, but it is also shallower than it looks. This post covers how the standard emerged, what a base-URL swap actually buys you, and the specific places where compatibility leaks, because the failures are rarely loud enough to show up in a demo.

How one vendor's API became the standard

No standards body chose the Chat Completions format. It won by being first to scale: by the time serious alternatives existed, thousands of applications, SDKs, and frameworks had already encoded its message array, its tools schema, and its server-sent-events streaming protocol. For every new provider, the cheapest path to those integrations was to speak the same format, and the ecosystem obliged from both directions.

On the provider side, compatibility endpoints are now table stakes. Anthropic ships an OpenAI SDK compatibility layer for the Claude API, Google exposes Gemini through an OpenAI-compatible endpoint, and DeepSeek's API uses the OpenAI format natively. Open-model hosts such as Together and DeepInfra serve hundreds of open-weight models behind the same shape. On the serving side, vLLM and Ollama expose OpenAI-compatible servers, which means a self-hosted Llama deployment answers the same requests as a hosted frontier model.

The result is an unusual situation: the interface is portable while everything behind it is not. The same twenty lines of SDK code can reach models from a dozen organizations, trained on different data, with different tokenizers, different safety systems, and different pricing. That is the promise and the trap in one sentence.

It is worth being precise about which API became the standard, because OpenAI itself has moved on. The company's newer Responses API is where its own feature development now lands, while the ecosystem's compatibility layers overwhelmingly implement Chat Completions as it stood around 2024. The de facto standard is therefore a snapshot: stable, widely spoken, and slowly diverging from the API of the vendor that created it. Client applications increasingly expose a toggle for which dialect to speak, and a surface that only understands one of them will meet clients that assume the other. For teams building against compatible endpoints, Chat Completions remains the lingua franca, and that is unlikely to change soon precisely because so much infrastructure has standardized on it.

What a base-URL swap actually buys

The core loop transfers cleanly. Chat messages, system prompts, temperature and sampling controls, streaming, and tool definitions all travel in the shared format, and for most applications that covers the majority of what the code does with a model. Swapping base_url and the API key genuinely does let you evaluate a Claude or Gemini model inside an application written against the OpenAI SDK, which is exactly what the compatibility layers were built for. Anthropic is explicit about the intended scope: its layer exists to test and compare model capabilities, with the native API recommended for production features such as prompt caching and citations.

The swap also buys optionality that compounds over time. An application written against the shared format can adopt a new provider the week it launches rather than after an integration project, and it can put a translation layer, a gateway, or a router in front of itself without touching call sites. We covered the mechanics of that migration in How to Route OpenAI SDK Calls Across Multiple Models; the short version is that the ecosystem's convergence on one format is what makes drop-in infrastructure possible at all.

What the swap does not buy is identical behavior, and the differences hide in the parameters rather than the happy path.

Where compatibility leaks

The pattern to internalize is that compatible endpoints tend to ignore what they do not support rather than reject it. Anthropic's documentation states that most unsupported fields are silently ignored, and the same is true across much of the ecosystem. Your request succeeds, your tests pass, and the behavior you asked for simply does not happen. These are the leaks worth checking before trusting a new endpoint with production traffic.

Sampling parameters do not mean the same thing. Temperature on OpenAI models ranges from 0 to 2; Anthropic's layer accepts values between 0 and 1 and caps anything higher. A temperature: 1.3 that produced usefully diverse output on one provider becomes 1.0 on another without a warning. Fields like seed, logprobs, presence_penalty, and frequency_penalty are ignored entirely by some layers, which matters if your evaluation harness depends on reproducible samples or your product tunes repetition behavior.

Tool calling transfers in shape, not in guarantees. The tools array and the returned tool_calls structure are well standardized, and agent frameworks rely on that. The guarantees underneath differ: Anthropic's layer ignores the strict parameter, so tool arguments are not promised to conform to your JSON schema, where OpenAI's structured outputs enforce it. Providers also differ on parallel tool calls and on the exact sequencing of streamed tool-call deltas. An agent loop that parses arguments with a strict validator will surface these differences as intermittent failures on one provider and never on another.

Structured output is the least portable feature. response_format with a JSON schema is enforced on OpenAI, ignored by Anthropic's compatibility layer (the native Claude API has its own structured outputs mechanism), and supported with provider-specific caveats elsewhere. If your pipeline depends on schema-valid JSON, this single field decides whether a provider swap is an afternoon or a rewrite, and it deserves a dedicated test. Our guide to structured outputs and JSON mode covers the per-provider mechanics.

System messages are restructured in transit. OpenAI permits system and developer messages anywhere in the conversation; Anthropic supports a single initial system message, so its layer hoists every system message to the front and concatenates them with newlines. Applications that inject mid-conversation instructions (a moderation notice, a tool result summary, a persona switch) will see those instructions moved, which changes what the model actually reads.

Token accounting is provider-relative. The usage block comes back in the same shape everywhere, but the numbers are produced by different tokenizers. The same prompt tokenizes to different counts on GPT, Claude, and Llama-family models, so budgets, truncation logic, and per-request cost estimates calibrated on one provider drift on another. Detailed sub-fields differ too: Anthropic's layer returns prompt_tokens_details permanently empty, which quietly breaks cached-token dashboards. We looked at how tokenizers differ in What Is Tokenization.

Errors and rate limits only rhyme. Error formats are kept consistent in structure, but the messages, the granularity of status codes, and the rate-limit headers differ. Retry logic tuned to one provider's 429 behavior, retry-after semantics, and error taxonomy will retry the wrong things, or fail to retry the right ones, when pointed elsewhere.

Model naming is its own hazard. A unified surface still needs model identifiers, and providers disagree about what a name pins. Aliases like -latest, dated snapshots, and quantized variants served under the parent model's name all mean the model behind a fixed string can change without your code changing. We documented that failure class in Silent Model Swaps, and it applies doubly on multi-provider surfaces, where the same open-weight model may be served at different precisions by different hosts.

The layers that provide one API

Once the wire format is shared, there are three ways to get an actual single surface over multiple LLM providers, and they answer different questions. A fuller treatment of the middle two is in LLM Gateway vs LLM Router.

The shim is the lightest commitment and the right tool for evaluation work. The gateway earns its place when many teams share provider accounts and someone has to own governance. The router matters when the question shifts from how to reach the model you chose to which model each request should reach, which is the question we built our routing layer around: one OpenAI-compatible endpoint where the model field can name a specific model, or can be left to per-request selection with the decision recorded and explainable.

There is a structural point underneath the taxonomy. A unified API that stops at format translation hands you a new obligation: you now hold accounts with several providers and must decide, per workload, which model to name. The format problem and the selection problem look similar from a distance, and solving the first tends to reveal the second.

What to verify before trusting an endpoint

Compatibility claims are cheap to make and uneven in practice, so a short verification pass on real traffic is worth more than any feature matrix. The checks that catch the most in our experience:

1. Round-trip a real tool call, then validate the returned arguments against your schema with a strict parser. This catches ignored strict flags and argument drift in one test.
2. Diff a streamed and non-streamed response for the same prompt, including the final usage chunk. Streaming implementations diverge most at the edges: tool-call deltas and where usage is reported.
3. Send your actual parameter set and compare outputs against a baseline at the same settings, because ignored parameters do not error. If you rely on seed, response_format, or penalties, test those specifically.
4. Compare usage numbers for identical prompts across providers before reusing token budgets or cost-per-request assumptions.
5. Force a failure (an invalid key, an oversized request, a burst past the rate limit) and check that your retry and backoff logic reads this provider's errors correctly.
6. Pin down what the model name means: whether it is a stable snapshot, a floating alias, and, for open-weight models, which precision the host actually serves.

An hour spent on this list converts silent behavioral differences into known, handled ones, which is the entire game with compatible endpoints.

The standard is real, and so are the edges

The convergence on one wire format is one of the quietly useful things to happen to LLM infrastructure: it made multi-provider architectures, gateways, and routing layers practical, and it reduced provider switching from an integration project to a configuration change. Treating every compatible endpoint as interchangeable, however, extends the standard further than it goes. The format is shared; the tokenizers, guarantees, parameter semantics, and failure behavior are not.

The practical posture is to write against the shared format, keep provider-specific assumptions out of application code, and verify each endpoint at the edges listed above before it takes production traffic. That discipline is also what makes the next layer possible: once your requests are portable, the remaining question is which model each one should reach, and that is a question worth answering per request rather than per config file.