Comparison
Busbar vs LiteLLM
Everybody evaluating Busbar knows LiteLLM, so this page is specific about what's actually different. It isn't a feature list. LiteLLM is genuinely good at what it is: an AI gateway, a Python-native router with huge provider coverage. Busbar is a different kind of tool: an AI control plane in a single Rust binary. It covers more than LLMs (chat, embeddings, images, audio, moderations, and rerank across every provider) and does more than route (failover, breakers, governance, observability). The difference shows up in three places: how protocols are handled, what happens when a provider fails, and what the layer itself costs you at runtime. Every claim about LiteLLM below is cited to their own documentation, and every claim about Busbar to ours.
The short version
| Busbar | LiteLLM | |
|---|---|---|
| Core format | Superset core; all six protocols first-class, no privileged one | OpenAI-shaped core [7]; native ingress built endpoint by endpoint [4] |
| Native ingress + load balancing | Every protocol routes through pools with failover | /v1/messages routes cross-provider [4]; Gemini/Bedrock native routes stay within their provider [9]; raw pass-throughs bypass fallbacks [5] |
| Bedrock SDK ingress | A signed boto3 client, load-balanced across non-Bedrock backends, response re-encoded as valid binary eventstream | boto3 /converse and /invoke load-balance across Bedrock deployments only; no cross-provider egress [9] |
| Failure classification | Fault-attributed: provider outage, auth/billing, client fault, and context-length each handled differently, per (pool, lane) | Exception hierarchy feeding retry counts and a cooldown timer [3] |
| Recovery | Single-flight half-open probe, exponential backoff with jitter, Retry-After honored as a floor | Deployment returns when cooldown_time expires [3] |
| Failover boundary | In-flight, before the first byte, across protocol families | Retry after the failed call returns [3] |
| Runtime | One static Rust binary; no database, no Valkey (Redis-protocol compatible). Optional drop-in Postgres/Valkey store plugins (1.5.0) for cluster-shared governance | Python/Uvicorn workers + PostgreSQL [2]; Redis for multi-instance state [3] |
| Added latency | 0.026 ms median, 0.033 ms p99 (small call); 0.084 ms median, 0.158 ms p99 translating a 12k-token body cross-protocol, from our published benchmark [6] | 2 ms median, 8 ms p95, 13 ms p99, from their published benchmark [1] |
| Memory | ~10.25 MB idle (v1.4.1), from our published numbers [8] | ~359 MB peak (Python), ~31.7 MB (Rust core, beta), from their published numbers [7] |
| Throughput | 156,223 req/s on 16 cores of a c7g.8xlarge (Graviton3), 9,750+ req/s per core, linear, from our published benchmark [6] | ~1,170 req/s from a 4-machine deployment, 4 vCPUs each (16 total, ~73 req/s per vCPU), from their published benchmark [1] |
| Provider coverage | 114 verified preconfigured, plus any provider on the six protocols (now including OAuth-authed backends like Google Vertex AI (Gemini + Claude) and Azure OpenAI via Entra ID) through your own providers.yaml, build a config | 100+ preconfigured out of the box |
| Observability | Prometheus, OTLP, admin API; bring the Grafana you already run | Bundled admin UI |
In plain terms, at the p99 tail: Busbar's added overhead is 33 µs against LiteLLM's 13 ms, about 394× less, and even translating a 12k-token body both ways it's still ~82× less. One Busbar vCPU handles about as many requests per second as 134 LiteLLM vCPUs (9,750+ req/s against ~73 req/s). Every ratio measures the same thing: how much CPU a request costs.
Benchmark figures are each project's own published numbers on its own hardware, so read the
latency and throughput rows as orders of magnitude, not a stopwatch. Busbar's harness is
checked into the repo under bench/, and the opt-in Server-Timing
header lets you measure Busbar's overhead on your own traffic instead of trusting either
table. Full methodology. Better still, don't trust either
vendor: onthebench.ai is a neutral benchmark that runs
every AI gateway through the identical test, Busbar and LiteLLM side by side, and anyone can
re-run it.
No privileged protocol
LiteLLM's core normalizes everything into the OpenAI format ("one OpenAI-compatible API,"
in their own words [7]), and its native-format ingress is built
endpoint by endpoint on top of that core. Where an endpoint has been built, it routes:
Anthropic /v1/messages reaches other providers [4],
while the Gemini and Bedrock native endpoints load-balance within their own provider
[9]. The raw pass-through routes forward without fallbacks
[5]. What you don't get, on any of them, is what the six-protocol
matrix gives by construction: any native ingress reaching any backend.
Busbar doesn't make you pick, on any protocol. All six wire protocols (OpenAI, Anthropic, Gemini, Bedrock, Cohere, Responses) are first-class in both directions, through one superset intermediate representation: an internal format rich enough to hold every protocol's features, which each protocol maps into and out of, so nothing is flattened to any vendor's shape. When ingress and egress match, the body passes through byte-for-byte. An OpenAI SDK and a signed Bedrock SDK can hit the same pool at the same time, each gets its own native response back, and failover works underneath both.
That Bedrock case is worth dwelling on, because it's the hardest one: Busbar verifies the
inbound AWS SigV4 signature, translates the Converse request through the IR to whichever
backend the pool selects, then re-encodes the streamed response as binary
application/vnd.amazon.eventstream frames with valid CRC32 checksums, so boto3
parses it as if Bedrock itself had answered. We haven't found this anywhere else.
How translation works.
What that means in practice: Busbar is valuable before the second provider exists. A team
locked to Bedrock today (a BAA that covers AWS but not yet Anthropic, say) points boto3's
endpoint_url at Busbar and changes nothing else; same-protocol traffic passes
through byte-for-byte, and no new entity enters the data path, because Busbar runs in your
infrastructure. The day a direct Anthropic BAA lands, an Anthropic lane is one config edit,
and the application code never learns it happened. With LiteLLM, Bedrock-SDK ingress
load-balances across Bedrock deployments only [9], so that same
migration starts with rewriting your integrations.
Six protocols, not a list to be on
The honest frame here isn't a provider count, it's protocols. Busbar speaks six wire protocols, and
nearly every hosted and local provider speaks one of them, so a provider isn't an entry you wait for us
to add: it's a few lines in your own providers.yaml: a name, a base_url, the
env var holding its key. OpenAI-compatible alone already reaches Groq, Mistral, Together, DeepSeek,
Fireworks, xAI, Ollama, vLLM, and most of the rest of anyone's list. Adding
one takes about a minute. There's no list to be on, and no ceiling on reach.
But if you want a number: Busbar ships 114 verified providers preconfigured, right
alongside LiteLLM's 100+. The real difference is what "more" costs. On Busbar, every provider beyond the
114 is three config lines (a name, a base_url, the env var holding its key)
- not code, and not a wait on us. The shipped catalog is curated rather than scraped
(each entry's error-code mappings feed the breaker's fault attribution), and LiteLLM's has more entries
typed for you; with Busbar you occasionally type that one line yourself. The only case
that ever needs new code (not config) is a provider whose wire format is none of the six and
compatible with none. Everything else is three lines away.
A real circuit breaker, not a cooldown timer
LiteLLM does classify errors: it maps provider failures into an exception hierarchy with per-exception retry policies [3]. The difference is what classification drives. There, it feeds retry counts and a cooldown timer; a cooled-down deployment returns when the timer expires. Here, it feeds a state machine with distinct dispositions:
- A provider outage or rate-limit storm trips the lane and cools it down, honoring the provider's
Retry-Afteras a floor. - An auth or billing failure is benched immediately with a sticky cooldown (30 minutes by default), because retrying a revoked key into a wall helps nobody.
- A client's bad request is relayed verbatim and never penalizes the lane, so a healthy backend is never ejected because a caller sent malformed input.
- A context-length overflow fails over to a larger-context lane without recording a fault.
Breaker state is per pool, per lane, so a lane rate-limiting in one pool can still serve another where it's healthy. Recovery is a single half-open probe request, not a thundering herd. And a failing request reroutes to the next lane before your client sees a single byte, even on a streaming request. No Valkey, no coordination layer, it's all in the binary; LiteLLM uses Redis to share cooldown state across multiple instances [3]. The full state machine.
The control plane itself is basically free
One static Rust binary, no interpreter, no garbage collector in the request path. Median added latency is in the tens of microseconds, and stays at ~84 µs median (158 µs p99) even translating a 12k-token body between protocols in both directions [6]. LiteLLM's own published benchmark reports 2 ms median and 13 ms p99 of proxy overhead [1]: different harness, different hardware, but a different order of magnitude too.
And you don't have to take either number on faith. Set one config flag
(observability.emit_server_timing: true) and Busbar reports its own overhead on
every response in a standard Server-Timing header, so you can check the number
on your own traffic instead of trusting our benchmark.
Benchmark and methodology.
Nothing else to run, until you choose to
Busbar's runtime footprint is the binary, full stop. Failover and breaker state live in the process, and governance (virtual keys, budgets, rate limits) enforces from in-memory counters by default, so a production deployment is one artifact and a YAML file. LiteLLM's production guide calls for PostgreSQL for virtual keys and spend tracking, and Redis as soon as you run more than one instance [2] [3]: two stateful services that exist before the first request is served.
With Busbar those services are optional, not structural. 1.5.0 adds drop-in Postgres and Valkey store plugins (signed archives Busbar verifies and loads; no recompile) for teams that want governance state durable or shared across a cluster. A single node needs neither, and the binary without them stays lean. Works standalone, scales when you ask it to.
About LiteLLM's Rust rewrite
LiteLLM is rebuilding its data plane in Rust, and their launch post says why in plain terms: "Under real load, CPU and memory climb with concurrency, and pods get OOM-killed at the worst time" [7]. We think they're right, and the move validates the architecture Busbar has shipped since day one: a compiled, no-GC data plane is what this job requires.
The differences that remain are worth stating precisely. Their Rust core is in early beta, with a staged rollout planned through December 2026 [7]; Busbar has been a single Rust binary since its first release, so there is no migration to wait for or live through. Their plan keeps customer plugins (auth, guardrails, callbacks) in an optional Python sidecar and long-tail providers on Python [7]; Busbar's request path is one process end to end. And a Rust rewrite of an OpenAI-normalized gateway is still an OpenAI-normalized gateway: the hub format, the endpoint-by-endpoint native ingress, and the pass-through split don't change with the implementation language. On the numbers, their published Rust-core overhead (~0.05 ms) lands in the same class as ours; memory (their 31.7 MB peak vs our ~10.25 MB idle) and the single-binary deployment story remain architectural, not implementation, differences [7].
Where LiteLLM is the right call
- You want an in-process Python router library, imported into your code rather than run as a network service. That's LiteLLM's home turf and the more natural fit.
- You want a bundled dashboard. LiteLLM ships an admin UI; Busbar exposes Prometheus, OTLP, and an admin API, and expects you to point the dashboards you already run at it.
- Deep Python-ecosystem integration matters to you (LangChain, LlamaIndex, and friends), though those frameworks also work with Busbar today: point their OpenAI or Anthropic client at Busbar's base URL and you keep your whole stack, with failover and translation underneath it.
If that's your shape, use LiteLLM. It's a good tool and we say so without hedging.
If this layer is production infrastructure carrying all of your AI traffic, you need a control plane, not just a gateway, and that's the problem Busbar was built for. Get started in five minutes, read why Busbar exists, or (already on LiteLLM?) here's how to migrate.
Already on LiteLLM? Convert your config in seconds → Paste yourconfig.yaml into the converter: models, fallbacks → pools, breaker settings, and keys map automatically. Runs entirely in your browser. References
Facts about LiteLLM checked against docs.litellm.ai on July 22, 2026. If something here is out of date, tell us and we'll fix it.
- LiteLLM benchmarks: 2 ms median / 8 ms p95 / 13 ms p99 "LiteLLM Overhead Duration"; ~1,170 RPS across 4 instances of 4 vCPU / 8 GB.
- LiteLLM production guide: Uvicorn workers, PostgreSQL, deployment guidance, Redis version requirements.
- LiteLLM routing and fallbacks: retry-based fallbacks,
allowed_fails,cooldown_time, per-exception retry policies, and Redis for tracking cooldown and usage state across instances. - LiteLLM
/v1/messages: Anthropic-format ingress routed through the LiteLLM router. - LiteLLM pass-through endpoints: raw native-format routes forwarded to their provider, without fallbacks.
- Busbar benchmark: methodology, per-protocol numbers, and the reproduction steps; public harness at GetBusbar/benchmarking.
- LiteLLM Rust launch post: the Python problems in their words, the Rust-core numbers, the beta status and staged rollout through December 2026, and the Python sidecar for plugins.
- Busbar, in numbers: memory, latency, and throughput measurements on the released binary, and how to reproduce them.
- LiteLLM Bedrock Converse endpoint and generateContent endpoint: routed native ingress that load-balances within its own provider's deployments.