Busbar

Scales to 156k+ req/s on 16 cores

One static binary. Linear with cores. Microseconds of overhead, single-digit-MB idle.

Throughput 9,750+ rps per core, linear · 156,223 on 16 cores
Added latency 33 µs busbar;dur p99, one request · 16-core c7g scaling run (v1.4.1)
Memory 10.25 MB resident, idle · one static binary (v1.4.1)

Add cores, add capacity, linearly

Busbar's request path is CPU-bound on the work that matters (parse, translate, serialize) and nothing else. No GC, no database in the hot path, no per-request lock. So throughput tracks the cores you give it, in a straight line. Measured on one c7g.8xlarge (32 vCPU Graviton3), driving unique requests through a fast upstream (so every request is real proxy work, not a cache hit):

req/s (scales linearly) added latency (flat)
050k100k150k 04080 246810121416 CPU cores → 2 cores · 19,838 req/s · 100% success · 58 MB · added latency 29/37 µs (p50/p99) 4 cores · 38,285 req/s · 100% success · 80 MB · added latency 26/33 µs (p50/p99) 6 cores · 61,505 req/s · 100% success · 94 MB · added latency 27/34 µs (p50/p99) 8 cores · 83,850 req/s · 100% success · 91 MB · added latency 26/32 µs (p50/p99) 10 cores · 101,853 req/s · 100% success · 102 MB · added latency 26/33 µs (p50/p99) 12 cores · 120,038 req/s · 100% success · 106 MB · added latency 26/33 µs (p50/p99) 14 cores · 142,038 req/s · 100% success · 110 MB · added latency 26/33 µs (p50/p99) 16 cores · 156,223 req/s · 100% success · 118 MB · added latency 26/33 µs (p50/p99) 2 cores · added latency 29 µs p50 / 37 µs p99 (busbar;dur, concurrency 1) 4 cores · added latency 26 µs p50 / 33 µs p99 (busbar;dur, concurrency 1) 6 cores · added latency 27 µs p50 / 34 µs p99 (busbar;dur, concurrency 1) 8 cores · added latency 26 µs p50 / 32 µs p99 (busbar;dur, concurrency 1) 10 cores · added latency 26 µs p50 / 33 µs p99 (busbar;dur, concurrency 1) 12 cores · added latency 26 µs p50 / 33 µs p99 (busbar;dur, concurrency 1) 14 cores · added latency 26 µs p50 / 33 µs p99 (busbar;dur, concurrency 1) 16 cores · added latency 26 µs p50 / 33 µs p99 (busbar;dur, concurrency 1) 156k req/s ~33 µs, flat
Throughput (green) climbs in a straight line, about 9,750+ req/s per core, while Busbar's own added latency (amber) stays flat at ~33 µs no matter the load: 2 cores 19k → 16 cores 156k, 100% success on one c7g.8xlarge. It stays flat until it runs out of CPU, not until it falls over (add cores and the ceiling moves up, never unstable), and Busbar reports that number in-band on every response (Server-Timing: busbar;dur), so you read it on your own traffic instead of trusting us.

How that stacks up

We stand on our own numbers, but here's the short version against the two alternatives people ask about most. Two separate comparisons to Busbar, not a ranking of Bifrost against LiteLLM. Figures we measured are tagged we measured; each project's own published figures their published.

vs LiteLLM (Rust) their published we measured matches on speed · wins on the rest · same-box bench Same speed, more gateway On a same-box run, latency and throughput are a tie. Both are compiled Rust. So LiteLLM's Rust beta (GA December 2026) gets you nothing you don't already get from Busbar. Same speed. But Busbar also gives you six-protocol translation both ways, any SDK routed to one compliant backend with zero rewrites, and governance in the compiled core. Theirs is a bare single Anthropic /v1/messages endpoint; the rest runs through the Python sidecar. Memory is the one gap, and it's Busbar's win: Busbar idles at 7.1796875 MB against LiteLLM Rust's ~253 MB (their published 21.8 MB config returns HTTP 400). Why pick the one that does less? The full comparison → vs Bifrost we measured throughput · memory · same box, real proxy ~9× the throughput Same box, same fast upstream, unique traffic (real proxy work, no cache). On 16 cores Busbar sustains 156,223 req/s where Bifrost does 18,346, about 9×, at a fraction of the memory. Both scale with cores; Busbar sustains ~9× the throughput per core on identical real-proxy traffic. The full head-to-head → vs LiteLLM (Python) their published p99 overhead · memory · their published specs ~394× lower p99 overhead LiteLLM's published p99 overhead is 13 ms. Busbar's is 33 µs (busbar;dur p99, 16-core c7g scaling run), at ~35× less memory and ~134× the throughput per vCPU. We take their number at their word. The full comparison →

Our benchmark: one clean box per gateway we measured

This is our own benchmark, public at github.com/GetBusbar/benchmarking. Every gateway gets its own AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway. A fresh, identical box for each one, so nothing from a neighbor's run bleeds into the next. Two throughput numbers plus a mock-ceiling guardrail, so no result is secretly the harness's own limit. Clone it and run it yourself. And if you'd rather not start from a vendor's page at all, onthebench.ai publishes the field neutrally: every AI gateway measured identically, side by side, re-runnable by anyone. The benchmark measures; this page argues from it.

GatewayAdded latency (end-to-end) p99Throughput (RPS)Memory idle / peak
BusbarBusbar 111 µs 48,236 7.18 / 314 MiB
LiteLLM · Rust 106 µs 46,187 252.832 / 258 MiB
agentgateway 214 µs 25,668 25.051 / 49 MiB
AISIX (api7) 256 µs 17,463 67.164 / 443 MiB
Helicone 319 µs 14,504 42.914 / 57 MiB
Kong 408 µs 20,575 404.996 / 618 MiB
APISIX 444 µs 21,091 178.848 / 210 MiB
Bifrost 904 µs 5,341 219.73 / 906 MiB
GoModel 2,023 µs 1,877 52.813 / 88 MiB
Portkey 3,511 µs 884 123.867 / 249 MiB
LiteLLM · Python 6,417 µs 186 1,081.781 / 1,106 MiB
TensorZero 40,993 µs measuring 48.734 / 72 MiB
Plano 218,982 µs measuring 638.199 / 966 MiB
One-API 1,556,903 µs measuring 87.648 / 145 MiB

Added latency = gateway p99 − direct-to-mock p99 (c1). Throughput = sustained RPS at a realistic 20 ms model delay. Memory = idle / peak RSS under load; did not serve = couldn't hold the endpoint. Measured 2026-07-31.

Busbar's own compute is 33 µs (busbar;dur p99, 16-core c7g.8xlarge scaling run). The benchmark's 111 µs is end-to-end p99 on the m7g.2xlarge box.
0 µsClient→ Busbar →AI
111 µsClient← Busbar ←AI

Green is Busbar's real work, timed on its own clock (busbar;dur), no wire in it. Grey is kernel + TCP time every proxy pays; co-locate Busbar with your app and it's sub-millisecond, and against an ~800 ms model call all of it vanishes.

we measured is our number on identical hardware; their published is that project's own figure, taken at their word.