What is LLM routing?

LLM routing is the choice of which provider, and sometimes which model, handles each request. A router weighs price, latency, uptime, and the features a request needs, then sends it to the best available option. Routing lets one application use many providers without hard-coding any of them.

Also called: Model routing, Provider routing · Last updated

Model routing and provider routing

Provider routing picks among the providers that serve the model you asked for. Model routing goes further and picks the model itself based on the request. Both aim for the same result: an acceptable answer at the lowest cost and delay.

Routing on LLM Gateway

Request a model by its ID and the gateway selects a provider that can handle your prompt size and parameters. Prefix the ID with a provider, such as openai/gpt-4o, to pin one.

Learn more

Related terms