Concurrent Request Limits and Overload Protection

A per-minute rate limit can't see duration. Six hundred requests per minute sounds bounded — until each one streams for two minutes and holds 1,200 connections open at once. Long-lived agentic streams pile up in a way RPM budgets never notice, and one tenant's runaway parallelism can exhaust shared gateway capacity for everyone else. Concurrent request limits close that gap: each organization now has one fleet-wide budget of in-flight requests across all inference endpoints — chat completions, messages, responses, embeddings, moderations, rerank, OCR, images, speech, transcriptions, videos, and the AI SDK surface.

How Slots Work

A slot is held for a request's full lifetime — including the entire duration of a streamed response — and freed the moment the response finishes or the connection closes. Slots free up continuously as requests complete, so there is no fixed window to wait out: a retry after a short backoff typically succeeds.

For pay-as-you-go organizations the ceiling scales with the same trust tier that already drives your per-minute limits and spend caps:

Requests over the budget are rejected with a retryable 429:

1HTTP/1.1 429 Too Many Requests2Retry-After: 13
4{5    "error": {6        "message": "Too many concurrent requests for this organization (limit: 100). Retry shortly, or reduce request concurrency.",7        "type": "rate_limit_error",8        "code": "rate_limit_exceeded"9    }10}

1HTTP/1.1 429 Too Many Requests2Retry-After: 13
4{5    "error": {6        "message": "Too many concurrent requests for this organization (limit: 100). Retry shortly, or reduce request concurrency.",7        "type": "rate_limit_error",8        "code": "rate_limit_exceeded"9    }10}

Unlike the per-minute limits, Enterprise organizations are not exempt — unbounded single-tenant concurrency exhausts shared capacity regardless of plan. They get the top-of-ladder ceiling instead, and can contact us to raise it further.

Overload Shedding (529)

The same release adds pod-level backpressure. When a gateway instance is momentarily at its own capacity, new inference requests are shed instantly with a 529 and Retry-After: 1 — matching Anthropic's 529 Overloaded — instead of queueing until the instance degrades. Health checks and /v1/models are never shed, so the rest of the fleet keeps absorbing traffic while the hot instance recovers.

Both responses carry Retry-After — respect it with exponential backoff and a spike passes through instead of failing.

Rate limits docs → | Contact us about Enterprise →