Architecture

This document traces a request end-to-end and explains the two seams that make Busbar’s thesis, protocols, not providers, work: the superset IR with its ProtocolReader / ProtocolWriter traits, and the two-stage failure-disposition pipeline.

Request lifecycle

1. Ingress & protocol detection

The route table (crates/busbar/src/main.rs build_router, crates/busbar/src/ingress/mod.rs) determines the ingress protocol by path, not by sniffing the body. All six protocols are first-class ingress, one handler per protocol (Gemini’s handler is reachable via two path prefixes, v1 and v1beta):

• POST /{name}/v1/messages → ingress anthropic. name is a model or a pool.
• POST /{provider}/{model}/v1/messages → ingress anthropic, ad-hoc direct route.
• POST /v1/chat/completions → ingress openai. The body’s model field names the model or pool.
• POST /v1/responses → ingress responses (OpenAI Responses API). Model in the body.
• POST /v2/chat → ingress cohere. Model in the body.
• POST /v1/models/{*rest} and POST /v1beta/models/{*rest} → ingress gemini. Both the stable v1 and the v1beta path prefixes are accepted by the same handler, because the google-generativeai / Gen AI SDKs use either surface. The model and the action (:generateContent / :streamGenerateContent) are packed into the last path segment after a :; axum can’t split on : inside a segment, so the tail is captured with a wildcard and split in gemini_ingress.
• POST /model/{model_id}/converse and /model/{model_id}/converse-stream → ingress bedrock. The model is in the path; the streaming variant is selected by the endpoint suffix.

This splits cleanly into body-model protocols (openai, responses, cohere, the model/pool lives in the request body) and path-model protocols (anthropic, gemini, bedrock: the model/pool lives in the URL). A small injection shim normalises both into the same internal model/pool selection so the rest of the pipeline is protocol-agnostic.

Management/observability routes (/stats, /healthz, /metrics, /api/v1/admin/keys...) are handled separately.

2. Authentication

auth_middleware (crates/busbar/src/auth/mod.rs) runs before routing:

• /healthz is always open (liveness probes must not require a token).
• /metrics is not exempted, Prometheus telemetry (lane/pool topology, per-protocol counters, error rates) is an information-disclosure surface, so it goes through the same auth check as any other route. It is gated by the data-plane auth chain (auth.chain): a request must satisfy some module in the chain (the built-in keys signed-token verifier, or an IdP auth module). With an empty chain (chain: []) the check admits unconditionally and /metrics is effectively open, so restrict it at the network layer if you need unauthenticated scraping.
• The admin API (/api/v1/admin/*) does not run on the data plane at all. It is served on a physically separate listener, admin_listen (default 127.0.0.1:8081, loopback), and gated by its own chain, admin_auth (default [admin-tokens]). An admin token arrives as Authorization: Bearer or X-Admin-Token; no valid admin credential means a 401. Because the socket is separate, a caller on the data port can never reach the control plane. Exposing admin_listen off loopback is a boot error unless you set admin_tls.client_ca (mTLS on the admin listener) or the explicit admin_insecure waiver (for operators fronting admin with their own mesh).
• On the data plane, the caller’s bearer token is threaded through the request. Whether Busbar signs the upstream call with its own lane key or forwards the caller’s credential is a separate config knob, upstream_credentials: (Own, the default, vs Passthrough), independent of which auth module ran at the front door. Under governance the resolved virtual key is attached for downstream ACL and budget checks.
• Bedrock ingress takes one of two paths. When the data-plane chain does not verify a caller (an empty chain, passthrough egress), extract_client_token reads only bearer-style carriers and ignores the SigV4 header, which is forwarded upstream (passthrough) or dropped. When governance is active, crates/busbar/src/auth/mod.rs verify_bedrock_sigv4 intercepts requests carrying Authorization: AWS4-HMAC-SHA256, verifies the full SigV4 signature plus body-hash integrity (x-amz-content-sha256), and on success attaches the resolved virtual key’s GovCtx so all governance checks apply. The AWS credential pair (aws_access_key_id
  • aws_secret_access_key) is minted via POST /api/v1/admin/keys with "issue_aws_credential": true. crates/busbar/src/sigv4.rs provides signing primitives; the inbound verifier lives in crates/busbar/src/auth/mod.rs.

3. Governance checks

When a virtual key is resolved, the route handler enforces, in order: allowed-pools (403), budget (429, or 400 for Bedrock ingress), and rate limits (429 + Retry-After) before forwarding. The budget check walks the key’s whole chain (the key’s own bucket, then its budget_group, then that group’s parent, up to the root) and admits only if every bucket is under cap; the 429 names which bucket blocked. Budget exhaustion does not emit 402: no upstream vendor returns 402 for an over-quota condition, so a 402 would be a router-side tell. Instead each ingress writer maps to its native quota shape: 429 (insufficient_quota) for OpenAI / Responses / Anthropic / Gemini / Cohere, and 400 (ServiceQuotaExceededException) for Bedrock. The flat per-request fee is charged at admission; the token counts land on the ledger when the response stream completes. Spend itself is never stored: it is derived at read time from the accumulated per-model tokens times the current top-level rate_card, so a rate correction re-prices past and present windows on the next read. See operations.md.

4. Pool / lane selection

For a pool target, forward_with_pool (crates/busbar/src/proxy/engine/mod.rs) selects a member:

1. Affinity preference: if a session header is present and the sticky member is usable, use it; otherwise fall through.
2. Exclusions: configured failover.exclusions and already-tried lanes (across failover hops) are removed from the candidate set.
3. SWRR: select_weighted (crates/busbar/src/store/mod.rs) runs Nginx-style smooth weighted round-robin over the usable candidates, using per-pool current_weight state. A lane is usable only if it isn’t dead, isn’t out of lifetime budget, and its breaker cell admits it.
4. Concurrency: the selected lane’s semaphore permit is acquired (a lane at its max_concurrent cap is skipped/awaited).

A direct/ad-hoc route is the degenerate case: a single-member candidate set of weight 1.

5. Cross-protocol translation (the IR seam)

If the ingress protocol differs from the selected lane’s protocol, Busbar translates the request through the superset IR:

ingress.reader().read_request(body)  →  IrRequest  →  lane.writer().write_request(ir)

The IR (crates/busbar/src/ir/mod.rs) is a superset of all six protocols’ representable content: system blocks, messages with text / thinking (+signature) / tool-use / tool-result / image blocks, tools (name + description + JSON schema), max_tokens, temperature (held as f64 so a caller’s value never silently mutates), a stream flag, and an extra passthrough map for fields outside the modeled subset (provider-specific sampling knobs with no first-class IR field, etc.). Same-protocol REQUESTS skip the IR entirely and pass through byte-for-byte, but only when the client named the lane’s exact wire model — a pool-alias route (e.g. model: "fast" resolving to a specific lane) rewrites the model and re-serializes instead. Same-protocol RESPONSES pass through byte-for-byte on the wire but still decode each frame through the IR as a usage side-channel (see docs/protocols.md’s “Same-protocol passthrough”); only the re-encode is skipped, not the IR round-trip.

ProtocolReader and ProtocolWriter (crates/busbar/src/proto/mod.rs) are the per-protocol edges:

• ProtocolReader: read_request (wire → IR), read_response / read_response_event(s) (wire → IR, with stateful fan-out for flat streams like OpenAI’s), and extract_error / classify (the breaker’s Stage 1).
• ProtocolWriter: write_request (IR → wire), write_response / write_response_event (IR → wire), rewrite_model, upstream_path[_for[_stream]], and the auth hooks: auth_headers(key) for static headers and sign_request(key, ctx) for per-request signing (overridden by Bedrock for SigV4). It also provides probe_body: a one-token request used by active health probes, so every protocol gets a valid probe for free.

A Protocol bundles a name + reader + writer; the ProtocolRegistry resolves them by name at startup. This is the entire reason a “provider” needs no code: any backend speaking a known protocol is just a catalog row.

6. Upstream auth & dispatch

The handler builds the upstream URL (base_url + the protocol’s path, or the provider’s path override), selects the key (lane key, or the caller’s key in passthrough mode), and computes auth via sign_request against a SigningContext (host, canonical URI, body, timestamp). For most protocols this is static headers; for Bedrock it computes AWS SigV4 with the region parsed from the host. The model field is rewritten to the selected lane’s model.

7. Two-stage failure disposition

Every non-2xx upstream response is run through a pipeline that decides who is at fault and therefore what to do (crates/busbar/src/proxy/engine/mod.rs, crates/busbar/src/breaker.rs):

Stage 1a  proto.reader().extract_error(status, body)  → RawUpstreamError
Stage 1b  normalize_raw_error(raw, provider.error_map) → CanonicalSignal (StatusClass)
Stage 2   classify_disposition(signal)                 → Disposition

Disposition is matched exhaustively (a project invariant: no _ => catch-all in breaker matches):

Disposition	Cause (StatusClass)	Lane effect	Request effect
ClientFault	client 4xx (400/404/422, context-aside)	none (tracked separately as client_fault)	relay verbatim to caller
TransientUpstream	5xx, timeout, network, overloaded, rate-limit	trip evaluation + cooldown (rate-limit honors Retry-After)	failover to next candidate
HardDown	billing/quota, auth (401/403)	lane marked dead (breaker trip)	auth → relay error to caller; billing → failover
ContextLength	context-length-exceeded	none (lane was healthy)	exclude ≤-context candidates, failover to a larger lane

This is the core correctness property: a healthy backend is never ejected because a caller sent a bad request. In passthrough mode, a 401/403 is the caller’s key failing, so it is relayed verbatim without touching lane health.

8. Response translation & usage accounting

On success, the response is streamed (SSE or Bedrock event-stream) or buffered:

• Same protocol: passthrough; native usage accounting and provider-specific fields survive untouched.
• Cross protocol: StreamTranslate (crates/busbar/src/proto/mod.rs) composes egress.reader().read_response_events with ingress.writer().write_response_event, re-framing each upstream event into the caller’s wire format. It reassembles frames split across chunks, threads stream decode state, decodes Bedrock’s binary application/vnd.amazon.eventstream on egress and re-encodes it (CRC32-valid frames) for Bedrock ingress, and emits the correct ingress terminator (data: [DONE] for OpenAI; Anthropic’s message_stop carries its own).

In both cases a usage tap reads token counts from the response (protocol-agnostic extraction across all six wire shapes), and, when governance is on, charges the resolved virtual key’s budget at stream completion. Failover is only possible before the first byte reaches the client; a mid-stream upstream failure records the breaker fault and emits a native error in the caller’s protocol, an SSE error event for SSE clients, a binary :message-type: exception frame for Bedrock-ingress (AWS eventstream) clients.

Circuit-breaker state

Breaker state is per-(pool, lane), stored in crates/busbar/src/store/mod.rs. The FSM is Closed → Open → HalfOpen → Closed, with exponential cooldown backoff and single-flight half-open probing. See operations.md for the full state machine, trip modes, and recovery behavior.

Observability hooks

Metrics are emitted at the ingress boundary (busbar_requests_total, the duration histogram) and at each upstream attempt/failure/trip/failover/translation (crates/busbar/src/metrics.rs, crates/busbar/src/proxy/engine/mod.rs). Optional OTLP spans and a request-log webhook are configured via the observability section.