What is a rate limit?

A rate limit is a cap on how much you can use an API in a period of time, usually counted in requests per minute or tokens per minute. When you exceed it the API returns HTTP 429 and refuses the request. Rate limits protect providers from overload and spread capacity across customers.

Also called: RPM, TPM, 429 error · Last updated

Handling a 429

Retry with exponential backoff and jitter, lower concurrency, and reduce tokens per request. If the limit comes from one provider, sending the request to another provider that serves the same model avoids the wait.

Two layers of limits

Each model provider enforces its own limits on your account. A gateway adds its own limits on top. LLM Gateway treats an upstream 429 as a provider failure and retries it on another provider when one is available.

Learn more

Related terms