Benchmark

“How much latency does Busbar add?” has one honest answer: its added latency, the microseconds Busbar itself spends parsing your request, translating it, and serializing the response, not the network and not the model. Busbar measures exactly that and reports it in-band on every response (when enabled; see below), in a standard header:

Server-Timing: busbar;dur=0.087

busbar;dur is total request time minus the upstream round-trip: everything Busbar did, nothing the upstream did. It’s a per-request number on Busbar’s own clock, so it is valid at the tail (each request is its own paired measurement) and independent of the upstream’s speed, which is what lets us measure the same number against an instant mock and against the real Anthropic API.

Every result below changes exactly one variable. If a benchmark moves two things at once, none of its numbers mean anything, so we don’t.

Throughput: it scales with cores, linearly

Busbar’s request path is CPU-bound on the work that matters (parse, translate, serialize) and nothing else (no GC, no database in the hot path, no per-request lock) so throughput tracks the cores you give it, in a straight line. On one c7g.8xlarge (32 vCPU Graviton3, no hyperthreading, so one vCPU is one real core), driving unique requests so every one is real proxy work (not a cache hit), with a dedicated load generator and mock on their own cores:

Busbar cores	2	4	8	12	16
req/s (100% success)	19,838	38,285	83,850	120,038	156,223
added latency (busbar;dur p99)	37 µs	33 µs	32 µs	33 µs	33 µs

Linear at ~9,750 req/s per core; added latency stays flat, because it’s per-request CPU work, independent of load. This sweep is a 1.4.0 run on a c7g.8xlarge (Graviton3); the per-request latency breakdown further down is a separate, earlier v1.0.0 run on a smaller c7g.2xlarge, so read the two as different measurements of the same binary family rather than one continuous benchmark. The full harness and the identical Bifrost sweep are public and reproducible in GetBusbar/benchmarking. Below is how we isolate that per-request cost itself.

Result

The setup (identical for every row)

The units, up front, so every number reads the same way:

• µs: microsecond. 1 µs = 1/1,000,000 of a second (and 1 ms = 1,000 µs). All latency is in µs.
• tok: token, the unit LLM APIs bill in (roughly 4 characters of text each). All payload sizes are in tok.
• p50 / p99 / p99.9: percentiles. p50 is the median (half are faster); p99 means 1 in 100 is slower; p99.9 is the tail, 1 in 1,000.

One AWS c7g.2xlarge (8 vCPU, Graviton3) in us-east-1, Amazon Linux 2023, running the released v1.0.0 aarch64 binary. Busbar is taskset-pinned to 2 dedicated cores; the load generator and the mock get different cores. The headline metric is Busbar’s own Server-Timing: busbar;dur= value, captured per request; Busbar’s CPU is logged from /proc. Each cell is thousands of requests at concurrency 1 (the per-request floor).

Result 1: added latency, swept by payload (Anthropic, vs mock)

Hold the protocol (Anthropic) and the upstream (instant mock) fixed; vary the payload. Two paths: a same-protocol passthrough, and a full cross-protocol translation (Anthropic in → OpenAI upstream → converted both ways). busbar;dur, in µs:

Path	Payload	p50	p99	p99.9
Same protocol (Anthropic→Anthropic)	~8 tok	38	45	51
	~1k tok	41	49	57
	~12k tok	93	137	147
Cross-protocol translate (Anthropic↔OpenAI)	~8 tok	48	55	62
	~1k tok	52	60	65
	~12k tok	84	158	174

It scales with bytes, not surprises. A tiny call is 38µs; a full cross-protocol translation of a ~12k-token (about 32 KB) body (parse, internal representation, re-serialize into a different protocol, both directions) is still 84µs. A fixed per-request floor plus the bytes Busbar parses and serializes; no allocation cliff. Busbar’s CPU stays ~0.5 of one core at concurrency 1.

The tail stays tight. Busbar is a single Rust binary with no garbage collector, so nothing in the request path pauses to sweep memory. Its p99.9 is only ~1.3 to 1.6× its p50 on small/medium payloads (and under ~2× even on the 12k-token translate); the added latency does not blow out into a tail. A garbage-collected proxy (Python/Node/JVM) pays an occasional GC pause on some requests, swelling p99/p99.9 by far more even when the median looks fine; we report p99.9 precisely so that’s visible.

Throughput, so “under one core” isn’t misread as a cap: saturating its 2 pinned cores Busbar sustained 19,505 req/s (~9,750 per core), and the Tokio runtime scales across cores.

Result 2: added latency by protocol (all six, vs mock)

Now hold the upstream (mock) and payload (~1k tok) fixed and vary only the wire protocol: same-protocol passthrough on each of Busbar’s six. This explains why it isn’t a single number: different formats cost different amounts to parse and serialize. busbar;dur p50 / p99, µs:

Protocol	p50	p99	note
OpenAI	37	43
Anthropic	42	49
Responses	37	44
Cohere	37	44
Gemini	40	47
Bedrock	63	76	+ SigV4 request signing

They cluster in the same tens-of-µs band; Bedrock is the outlier because it’s the one protocol that cryptographically signs every upstream request (AWS SigV4, an HMAC-SHA256 chain over the body), which is real CPU the others don’t pay. That’s an honest cost of speaking Bedrock natively, surfaced rather than hidden.

Result 3: invisible in production (paired, vs real Anthropic)

Change the upstream to the real api.anthropic.com and measure it honestly: direct and through-Busbar requests are interleaved (direct, Busbar, direct, Busbar, …) in the same window, so Anthropic’s minute-to-minute drift hits both equally and cancels in the delta. Small ~8 tok calls to claude-haiku-4-5, median client wall-clock:

Path (paired, interleaved)	p50 wall-clock
Direct (client → Anthropic)	807ms
Through Busbar (client → Busbar → Anthropic)	790ms
Difference	−17ms

Through Busbar was 17ms faster at the median. That of course doesn’t mean Busbar speeds up Anthropic. It means Busbar’s cost is below the provider’s own run-to-run noise: in a paired test where drift cancels, the difference lands on either side of zero. The interleaving is the whole point. Measure the two sides in separate passes and the provider’s own minute-to-minute drift masquerades as Busbar overhead; alternate them request-by-request and that drift hits both equally and subtracts out. We report only the median: at p99 the paired sample (n≈150 real calls) is dominated by provider-tail jitter, not Busbar.

And on those same requests, Busbar’s own self-reported number: busbar;dur p50 = 84µs. That’s the actual compute, about 0.01% of a ~790ms model call, and you can read it on your own traffic.

The one thing we won’t hide: Busbar is a proxy, so it adds one network hop (client to Busbar to provider). Here client and Busbar are co-located, so that hop is sub-millisecond and lost in the noise above; co-locate Busbar with your app in production and it stays sub-millisecond. We don’t claim zero. We claim small, and measured both ways.

Reproduce

One self-cleaning script on one named instance type; the upstream is the only thing that changes between Result 1, Result 2, and Result 3.

1. Launch a c7g.2xlarge (Graviton3, arm64), Amazon Linux 2023, us-east-1.

2. Build the released binary, stand up a tiny instant mock that speaks all six protocols, pin the cores:

curl -fsSL https://getbusbar.com/install.sh | sh   # or build the tagged source
# mock: a small Go HTTPS server returning a valid response for each of the six wire formats,
#       sized to the request's max-tokens; served over a real Let's Encrypt cert on a loopback host
#       (127-0-0-1.local-ip.sh → 127.0.0.1) so the release binary's public-CA-only upstream TLS trusts it.
taskset -c 0,1 ./busbar ... &     # Busbar -> 2 dedicated cores
taskset -c 6,7  ./mock  ... &     # mock   -> its own cores

3. Fire requests pinned to the remaining cores and read the Server-Timing: busbar;dur= header off each response; that is the metric. For Result 3, interleave direct and through-Busbar requests so provider drift cancels. The full self-contained script (launch → pin → measure → log CPU → tear down) is published with the release.

Why these choices

• Server-Timing, not a client stopwatch: Busbar’s own clock excludes the upstream round-trip and the client’s jitter, and it’s a W3C-standard header you can read in DevTools or any APM tool, on your own production traffic.
• A fixed cloud instance, pinned cores, logged CPU: reproducible by anyone; the number is the cost of one path of work, not a big box. “I used my MacBook” is not a citable number.
• Interleaved for the real-provider comparison: the only way to separate Busbar’s cost from a provider whose own latency drifts minute to minute.

The head-to-head method (comparing against another gateway)

The section above measures Busbar’s own added latency. Comparing against another gateway is a different question, and it has its own rule: change exactly one thing. Same machine, same mock upstream, same load tool, same rates: the only variable is which gateway is in the path. If a comparison moves two things at once, its numbers mean nothing.

We run the whole field, neutrally, in the open. That method lives in GetBusbar/benchmarking, a public harness that runs every gateway (Busbar and a dozen others) the same way: one fresh, identical Graviton box per gateway, pinned to the same cores, fronting the same fast mock, gated on the same error/latency thresholds. It takes no side: gateways are ordered by measured result, colored by implementation language, and any gateway that can’t hold the endpoint is recorded, not hidden. The full standings (added latency, sustained throughput, and memory for all of them) are on the performance page. Clone the harness and reproduce any row yourself. The rest of this section is how a single pairwise comparison is kept honest.

The mock is ours, fast, and the same for everyone. We benchmark gateways, not mocks. A slow mock bounds every gateway to the mock’s speed, which hides the gateway’s real cost and flatters the slower one; a fast (~0 ms) mock is never the bottleneck, so what’s left in the number is the gateway itself. We put the same fast mock in front of each gateway, and we don’t inherit anyone else’s mock as our ceiling.

How we treat a competitor’s published numbers, believe the unflattering, verify the incredible:

• If a competitor’s own published number is unflattering, we take it at their word. There’s nothing to learn by spending a machine to prove a competitor is slow. That’s why the LiteLLM comparison uses their published overhead and throughput as-is, clearly labelled as theirs.
• If a published number is too good to be true, we don’t accept it: we reproduce it. Bifrost publishes an “11 µs overhead” that their own benchmark tool never emits and their own /metrics can’t resolve (histogram buckets start at 5 ms). So for the Bifrost comparison we ran their own benchmark (their tool, their config, their mocker, co-located exactly as their docs prescribe) and report what it actually produces. If Bifrost tells us how the 11 µs is instrumented, we’ll run it on the same box and publish what we get.

Reproduce the Bifrost head-to-head yourself: their tool, their documented setup:

# One t3.xlarge (4 vCPU), CpuCredits unlimited. Co-located: mock + gateway + load tool on one box.
git clone https://github.com/maximhq/bifrost-benchmarking && cd bifrost-benchmarking
( cd mocker && go run main.go -port 8000 )        # their 0 ms mocker
go build benchmark.go                              # their load tool

# Gateway A — Bifrost, their documented t3.xlarge config (initial_pool_size 15000 / buffer_size 20000),
#             OpenAI provider pointed at the mocker (network_config.base_url = http://127.0.0.1:8000):
docker run --network host --pid host maximhq/bifrost:v1.6.4
./benchmark -provider bifrost -rate 1000 -duration 300 -output bifrost_1000.json
./benchmark -provider bifrost -rate 3000 -duration 180 -output bifrost_3000.json

# Gateway B — swap ONLY the gateway. Busbar release binary, openai provider → http://127.0.0.1:8000,
#             governance + logging off. Same tool, same rates, same box, same mock:
busbar &   # BUSBAR_CONFIG pointing at the mock; then re-run the same ./benchmark lines.

Subtract nothing: the tool reports end-to-end latency, memory, and success rate per run. The raw results for our runs are public in GetBusbar/benchmarking. Every number on the comparison pages reads from one version-stamped source (facts.ts); re-run on a new release, update facts.ts, and the pages re-derive.

Why a standard header. Server-Timing is W3C-standard, so you can read busbar;dur in browser DevTools or any APM tool, on your own production traffic, with no special tooling. The overhead number isn’t a one-time lab claim; it’s something you can verify continuously, on your own requests.

Enabling it. The Server-Timing header is opt-in (default off) to avoid fingerprinting Busbar on unauthenticated responses. Set observability.emit_server_timing: true in config.yaml to turn it on.