Back to blog

Ranked #1 on an Independent AI Gateway Benchmark

In the August 7 run of computesdk's independent AI gateway benchmark, LLM Gateway ranked first of six gateways with a composite score of 90.8 — driven by the tightest tail latency in the field, not the fastest median. Here's what the benchmark measures, what our numbers actually show, and how to run it yourself.

A circuit board with a glowing stopwatch on the central chip surrounded by a podium and rising bar chart, representing AI gateway latency benchmarks

Every AI gateway sits between your application and the model provider, and every one of them adds a hop. So the question any platform team should ask before adopting one is blunt: how much does that hop cost, and how predictable is it?

Vendor-published numbers are close to worthless here, because the vendor picks the conditions. An independent AI gateway benchmark is worth more — and one exists. computesdk runs an open-source harness that measures six gateways against a direct-to-provider control, on the same model, from the same machine, in the same round. In the August 7 run, LLM Gateway ranked first with a composite score of 90.8.

The more useful part isn't the ranking. It's which axis produced it.

What the AI Gateway Benchmark Measures

Every participant is pointed at the same model — Claude Haiku 4.5 — from a 4 vCPU / 16 GB runner in Northern Virginia. Each gateway runs 20 cold iterations (fresh connection: DNS + TCP + TLS + time to first token) and 20 warm iterations (pooled connection, TTFT only), round-robin, so no gateway gets a quieter stretch of network than another. The August 7 run recorded zero failed requests across the entire field.

The composite score blends cold-start latency, warm latency, and throughput, weighting both medians and 95th percentiles.

Rank Gateway Composite
1 LLM Gateway 90.8
— Anthropic (direct) 89.8
2 Concentrate AI 89.5
3 Pydantic 89.3
4 Vercel 89.0
5 OpenRouter 88.3
6 Cloudflare 88.1

Anthropic (direct) is a no-gateway control, not a competing product. It's there to isolate how much latency each gateway adds on top of the underlying provider.

Medians Hide the Problem. Tails Don't.

Look only at typical requests and every gateway looks the same. Fifty-four milliseconds covers our cold E2E median, the next-best gateway's, and the no-gateway baseline. That's noise for most workloads.

The percentiles are where the field separates:

Metric LLM Gateway Best of the rest No-gateway baseline
Warm TTFT p95 1151 ms 1289 ms 2035 ms
Cold E2E p95 1262 ms 1266 ms 1374 ms
Cold E2E median 594 ms 601 ms 648 ms

A warm p95 of 1151 ms against a field that ranges to 4329 ms is the difference between a slow request your users notice and one they don't.

This matters more than it used to. A chatbot turn is one call, and a bad p95 costs one person one pause. An agent turn fans out into many sequential calls, and the tail compounds: at ten calls per turn, a p95 stall stops being an edge case and starts being most turns. Tail latency is the number that decides whether an agentic workload feels responsive.

What We Changed in the Upstream Path

Our provider requests used to go out through bare global fetch. Undici's defaults close idle upstream sockets after four seconds and re-resolve DNS on every new connection, so any request arriving after a lull paid a full DNS + TCP + TLS handshake before its first token. Under Kubernetes' default ndots:5, one dropped UDP packet during the search-domain walk stalls a request for seconds — which is exactly the shape of a bad p95.

The gateway now installs a tuned dispatcher at boot (#3225):

  • 60-second idle keep-alive on upstream connections, so warm sockets stay warm through normal traffic gaps
  • An in-process DNS cache, so provider hostnames aren't re-resolved per connection
  • Redis auto-pipelining, batching the dozens of pre-dispatch commands each request issues into single round trips
  • Cache-mirror bookkeeping moved off the critical path, since no request result depends on it

None of that is exotic. It's the unglamorous work of making sure the connection to the provider is already open when your request shows up.

Read the Caveats Before You Cite This

This is one run. Latency benchmarks are noisy, the field is tightly packed — under three composite points separate first from last — and week-to-week ordering moves. The repository's public history includes runs where we placed lower, and we'd rather you find that from us than discover it yourself and wonder what else we left out.

Two things this result does not say. It does not say a gateway is faster than talking to the provider directly; our warm TTFT median is 629 ms against the baseline's 615 ms, and a hop is still a hop. And it does not say any particular change caused the week-over-week movement — the dispatcher work above was already deployed for earlier runs.

What the August 7 numbers support is narrower and, we think, more useful: under identical conditions, our slow requests were the least slow in the field.

Run It Yourself

The benchmark is open source and vendor-neutral. The harness, the scoring weights, and every historical run live in the repository:

1git clone https://github.com/computesdk/benchmarks2cd benchmarks && git checkout 7548e7584940 && pnpm install3
4export LLMGATEWAY_API_KEY=...   # from your dashboard5export ANTHROPIC_API_KEY=...    # the no-gateway control6
7pnpm bench:ai-gateway --iterations 20

7548e7584940 is the commit holding the August 7 results. --iterations 20 matches that run's 20 cold and 20 warm probes per gateway; the default is 10. Any gateway whose API key isn't set is skipped, so the command above measures us against the direct-to-Anthropic control — add the other gateways' keys to reproduce the full leaderboard.

Results land in results/ai-gateway/, and the composite weighting is in benchmarks/ai-gateway/scoring.ts.

Frequently Asked Questions

What does an AI gateway benchmark actually measure?

A good one measures the latency the gateway itself adds, isolated from the model provider's own speed. computesdk's harness does this by running a direct-to-provider control alongside the gateways, pointing everything at the same model, and running all participants round-robin so they share the same network conditions. It separates cold requests (fresh connection, including DNS, TCP, and TLS) from warm ones (pooled connection), because those fail in different ways.

Does an AI gateway add latency compared to calling the provider directly?

Yes. A gateway is an extra network hop, and no routing layer makes that free — in the August 7 run our warm TTFT median was 629 ms against the direct baseline's 615 ms. The question worth asking is what you get for it: failover across providers, unified billing and usage analytics, and a single API across model vendors. And the overhead is small enough that connection handling dominates it, which is why the tail, not the median, is where gateways actually differ.

How do I run the AI gateway benchmark myself?

Clone computesdk/benchmarks, set an API key for each gateway you want to include, and run pnpm bench:ai-gateway. Gateways without keys are skipped, so you can benchmark just your own stack against the direct-to-provider control. Results are written to results/ai-gateway/ as JSON, including every individual iteration, so you can compute your own percentiles rather than trusting anyone's composite.

Try LLM Gateway free — one API across every major provider, with automatic failover.