# LLM Gateway — Full Documentation

> LLM Gateway is an open-source, OpenAI-compatible API gateway for routing, managing, and analyzing requests across LLM providers. Use one API key, track usage and cost, configure caching and guardrails, and self-host or use the managed cloud. Current models and pricing: https://llmgateway.io/models

API base URL: https://api.llmgateway.io/v1 · Docs: https://docs.llmgateway.io · Site: https://llmgateway.io

This file concatenates the full text of every documentation page below.

# Introduction to LLM Gateway
URL: https://docs.llmgateway.io/


LLM Gateway is an open-source API gateway that sits between your applications and LLM providers like OpenAI, Anthropic, Google AI Studio, and more. It provides a unified, OpenAI-compatible API interface with built-in cost tracking, caching, and intelligent routing.

## How it works [#how-it-works]

Point your existing SDK at `https://api.llmgateway.io/v1` (or your self-hosted instance) and authenticate with an LLM Gateway API key — no code rewrites. The gateway routes each request to the right provider, tracks tokens and cost per model, provider, project, and API key, and fails over to a healthy provider when one errors. Pay per-token with prepaid credits at provider list rates, or bring your own provider keys (BYOK) for free.

## Why use a gateway? [#why-use-a-gateway]

* **One integration** — switch models or providers by changing a model string, not your code.
* **Cost visibility** — usage analytics and cost breakdowns across every provider in one dashboard.
* **Reliability** — automatic provider failover and response caching in front of every request.
* **No lock-in** — open source (AGPLv3), self-hostable, and OpenAI-compatible end to end.

## Features [#features]

All features are documented under https://docs.llmgateway.io/features; each feature page is included in full in this file.

## AI Tooling [#ai-tooling]

LLM Gateway is built to work seamlessly with AI agents and development tools.

AI tooling: https://docs.llmgateway.io/llms.txt (docs index for LLMs), https://docs.llmgateway.io/llms-full.txt (this file), https://docs.llmgateway.io/developers/mcp (MCP server), https://docs.llmgateway.io/guides/agent-skills (agent skills), and https://llmgateway.io/templates (templates and agents).

## Next Steps [#next-steps]

* [**Quickstart**](https://docs.llmgateway.io/quick-start) — Get up and running in minutes
* [**Overview**](https://docs.llmgateway.io/overview) — Learn more about what LLM Gateway offers
* [**Self-Host**](https://docs.llmgateway.io/self-host) — Deploy on your own infrastructure


# Overview
URL: https://docs.llmgateway.io/overview


LLM Gateway is an open-source API gateway for Large Language Models (LLMs). It acts as a middleware between your applications and various LLM providers, allowing you to:

* Route requests to multiple LLM providers (OpenAI, Anthropic, Google AI Studio, and others)
* Manage API keys for different providers in one place
* Track token usage and costs across all your LLM interactions
* Analyze performance metrics to optimize your LLM usage

## Analyzing Your LLM Requests [#analyzing-your-llm-requests]

LLM Gateway provides detailed insights into your LLM usage:

* **Usage Metrics**: Track the number of requests, tokens used, and response times
* **Cost Analysis**: Monitor spending across different models and providers
* **Performance Tracking**: Identify patterns and optimize your prompts based on actual usage data
* **Breakdown by Model**: Compare different models' performance and cost-effectiveness

All this data is automatically collected and presented in an intuitive dashboard, helping you make informed decisions about your LLM strategy.

## Getting Started [#getting-started]

Using LLM Gateway is simple. Just swap out your current LLM provider URL with the LLM Gateway API endpoint:

```bash
curl -X POST https://api.llmgateway.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -d '{
  "model": "gpt-4o",
  "messages": [
    {"role": "user", "content": "Hello, how are you?"}
  ]
}'
```

LLM Gateway maintains compatibility with the OpenAI API format, making migration seamless.

## Hosted vs. Self-Hosted [#hosted-vs-self-hosted]

You can use LLM Gateway in two ways:

* **Hosted Version**: For immediate use without setup, visit [llmgateway.io](https://llmgateway.io) to create an account and get an API key.
* **Self-Hosted**: Deploy LLM Gateway on your own infrastructure for complete control over your data and configuration.

The self-hosted version offers additional customization options and ensures your LLM traffic never leaves your infrastructure if desired.


# Quickstart
URL: https://docs.llmgateway.io/quick-start


Welcome to **LLM Gateway**—a single drop‑in endpoint that lets you call today’s best large‑language models while keeping **your existing code** and development workflow intact.

> **TL;DR** — Point your HTTP requests to `https://api.llmgateway.io/v1/…`, supply your `LLM_GATEWAY_API_KEY`, and you’re done.

***

## 1 · Get an API key [#1--get-an-api-key]

1. Sign in to the dashboard.
2. Create a new Project → *Copy the key*.
3. Export it in your shell (or a `.env` file):

```bash
export LLM_GATEWAY_API_KEY="llmgtwy_XXXXXXXXXXXXXXXX"
```

***

## 2 · Pick your language [#2--pick-your-language]

<Tabs
  groupId="language"
  items="[
'cURL',
'TypeScript',
'React',
'Next.js',
'Python',
'Java',
'Rust',
'Go',
'PHP',
'Ruby']"
>
  <Tab value="cURL">
    ```bash
    curl -X POST https://api.llmgateway.io/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
      -d '{
        "model": "gpt-4o",
        "messages": [
          {"role": "user", "content": "Hello, how are you?"}
        ]
      }'
    ```
  </Tab>

  <Tab value="TypeScript">
    ```typescript
    const response = await fetch("https://api.llmgateway.io/v1/chat/completions", {
    	method: "POST",
    	headers: {
    		"Content-Type": "application/json",
    		Authorization: `Bearer ${process.env.LLM_GATEWAY_API_KEY}`,
    	},
    	body: JSON.stringify({
    		model: "gpt-4o",
    		messages: [{ role: "user", content: "Hello, how are you?" }],
    	}),
    });

    if (!response.ok) {
    	throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();
    console.log(data.choices[0].message.content);
    ```
  </Tab>

  <Tab value="React">
    ```tsx
    import { useState } from "react";

    function ChatComponent() {
    	const [response, setResponse] = useState("");
    	const [loading, setLoading] = useState(false);

    	const sendMessage = async () => {
    		setLoading(true);
    		try {
    			const res = await fetch("https://api.llmgateway.io/v1/chat/completions", {
    				method: "POST",
    				headers: {
    					"Content-Type": "application/json",
    					Authorization: `Bearer ${process.env.REACT_APP_LLM_GATEWAY_API_KEY}`,
    				},
    				body: JSON.stringify({
    					model: "gpt-4o",
    					messages: [{ role: "user", content: "Hello, how are you?" }],
    				}),
    			});

    			if (!res.ok) {
    				throw new Error(`HTTP error! status: ${res.status}`);
    			}

    			const data = await res.json();
    			setResponse(data.choices[0].message.content);
    		} catch (error) {
    			console.error("Error:", error);
    		} finally {
    			setLoading(false);
    		}
    	};

    	return (
    		<div>
    			<button onClick={sendMessage} disabled={loading}>
    				{loading ? "Sending..." : "Send Message"}
    			</button>
    			{response && <p>{response}</p>}
    		</div>
    	);
    }

    export default ChatComponent;
    ```
  </Tab>

  <Tab value="Next.js">
    ```typescript
    // app/api/chat/route.ts
    import { NextRequest, NextResponse } from "next/server";

    export async function POST(request: NextRequest) {
    	const { message } = await request.json();

    	const response = await fetch(
    		"https://api.llmgateway.io/v1/chat/completions",
    		{
    			method: "POST",
    			headers: {
    				"Content-Type": "application/json",
    				Authorization: `Bearer ${process.env.LLM_GATEWAY_API_KEY}`,
    			},
    			body: JSON.stringify({
    				model: "gpt-4o",
    				messages: [{ role: "user", content: message }],
    			}),
    		},
    	);

    	if (!response.ok) {
    		return NextResponse.json(
    			{ error: "Failed to get response" },
    			{ status: response.status },
    		);
    	}

    	const data = await response.json();

    	return NextResponse.json({
    		message: data.choices[0].message.content,
    	});
    }

    // Usage in component:
    // const response = await fetch('/api/chat', {
    //   method: 'POST',
    //   headers: { 'Content-Type': 'application/json' },
    //   body: JSON.stringify({ message: 'Hello, how are you?' })
    // });
    ```
  </Tab>

  <Tab value="Python">
    ```python
    import requests
    import os

    response = requests.post(
        'https://api.llmgateway.io/v1/chat/completions',
        headers={
            'Content-Type': 'application/json',
            'Authorization': f'Bearer {os.getenv("LLM_GATEWAY_API_KEY")}'
        },
        json={
            'model': 'gpt-4o',
            'messages': [
                {'role': 'user', 'content': 'Hello, how are you?'}
            ]
        }
    )

    response.raise_for_status()
    print(response.json()['choices'][0]['message']['content'])
    ```
  </Tab>

  <Tab value="Java">
    ```java
    import java.net.http.HttpClient;
    import java.net.http.HttpRequest;
    import java.net.http.HttpResponse;
    import java.net.URI;

    String apiKey = System.getenv("LLM_GATEWAY_API_KEY");
    String requestBody = """
    {
      "model": "gpt-4o",
      "messages": [
        {"role": "user", "content": "Hello, how are you?"}
      ]
    }
    """;

    HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create("https://api.llmgateway.io/v1/chat/completions"))
        .header("Content-Type", "application/json")
        .header("Authorization", "Bearer " + apiKey)
        .POST(HttpRequest.BodyPublishers.ofString(requestBody))
        .build();

    HttpResponse<String> response = HttpClient.newHttpClient()
        .send(request, HttpResponse.BodyHandlers.ofString());

    System.out.println(response.body());
    ```
  </Tab>

  <Tab value="Rust">
    ```rust
    use reqwest::Client;
    use serde_json::json;
    use std::env;

    #[tokio::main]
    async fn main() -> Result<(), Box<dyn std::error::Error>> {
        let client = Client::new();
        let api_key = env::var("LLM_GATEWAY_API_KEY")?;

        let response = client
            .post("https://api.llmgateway.io/v1/chat/completions")
            .header("Content-Type", "application/json")
            .header("Authorization", format!("Bearer {}", api_key))
            .json(&json!({
                "model": "gpt-4o",
                "messages": [
                    {"role": "user", "content": "Hello, how are you?"}
                ]
            }))
            .send()
            .await?;

        let result: serde_json::Value = response.json().await?;
        println!("{}", result["choices"][0]["message"]["content"]);
        Ok(())
    }
    ```
  </Tab>

  <Tab value="Go">
    ```go
    package main

    import (
        "bytes"
        "encoding/json"
        "fmt"
        "net/http"
        "os"
    )

    type ChatRequest struct {
        Model    string    `json:"model"`
        Messages []Message `json:"messages"`
    }

    type Message struct {
        Role    string `json:"role"`
        Content string `json:"content"`
    }

    func main() {
        apiKey := os.Getenv("LLM_GATEWAY_API_KEY")

        requestBody := ChatRequest{
            Model:    "gpt-4o",
            Messages: []Message{{Role: "user", Content: "Hello, how are you?"}},
        }

        jsonData, _ := json.Marshal(requestBody)

        req, _ := http.NewRequest("POST", "https://api.llmgateway.io/v1/chat/completions", bytes.NewBuffer(jsonData))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Authorization", "Bearer "+apiKey)

        client := &http.Client{}
        resp, _ := client.Do(req)
        defer resp.Body.Close()

        fmt.Println("Response received")
    }
    ```
  </Tab>

  <Tab value="PHP">
    ```php
    <?php
    $apiKey = $_ENV['LLM_GATEWAY_API_KEY'];

    $data = [
        'model' => 'gpt-4o',
        'messages' => [
            ['role' => 'user', 'content' => 'Hello, how are you?']
        ]
    ];

    $options = [
        'http' => [
            'header' => [
                'Content-Type: application/json',
                'Authorization: Bearer ' . $apiKey
            ],
            'method' => 'POST',
            'content' => json_encode($data)
        ]
    ];

    $context = stream_context_create($options);
    $response = file_get_contents(
        'https://api.llmgateway.io/v1/chat/completions',
        false,
        $context
    );

    if ($response === FALSE) {
        throw new Exception('Request failed');
    }

    $result = json_decode($response, true);
    echo $result['choices'][0]['message']['content'];
    ?>
    ```
  </Tab>

  <Tab value="Ruby">
    ```ruby
    require 'net/http'
    require 'json'
    require 'uri'

    uri = URI('https://api.llmgateway.io/v1/chat/completions')
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true

    request = Net::HTTP::Post.new(uri)
    request['Content-Type'] = 'application/json'
    request['Authorization'] = "Bearer #{ENV['LLM_GATEWAY_API_KEY']}"

    request.body = {
      model: 'gpt-4o',
      messages: [
        { role: 'user', content: 'Hello, how are you?' }
      ]
    }.to_json

    response = http.request(request)

    if response.code != '200'
      raise "HTTP Error: #{response.code}"
    end

    result = JSON.parse(response.body)
    puts result['choices'][0]['message']['content']
    ```
  </Tab>
</Tabs>

***

## 3 · SDK integrations [#3--sdk-integrations]

```ts title="ai-sdk.ts"
import { llmgateway } from "@llmgateway/ai-sdk-provider";
import { generateText } from "ai";

const { text } = await generateText({
	model: llmgateway("gpt-4o"),
	prompt: "Write a vegetarian lasagna recipe for 4 people.",
});
```

```ts title="vercel-ai-sdk.ts"
import { createOpenAI } from "@ai-sdk/openai";

const llmgateway = createOpenAI({
	baseURL: "https://api.llmgateway.io/v1",
	apiKey: process.env.LLM_GATEWAY_API_KEY!,
});

const completion = await llmgateway.chat({
	model: "gpt-4o",
	messages: [{ role: "user", content: "Hello, how are you?" }],
});

console.log(completion.choices[0].message.content);
```

```ts title="openai-sdk.ts"
import OpenAI from "openai";

const openai = new OpenAI({
	baseURL: "https://api.llmgateway.io/v1",
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const completion = await openai.chat.completions.create({
	model: "gpt-4o",
	messages: [{ role: "user", content: "Hello, how are you?" }],
});

console.log(completion.choices[0].message.content);
```

***

## 4 · Going further [#4--going-further]

* **Streaming**: pass `stream: true` to any request—Gateway will proxy the event stream unchanged.
* **Monitoring**: Every call appears in the dashboard with latency, cost & provider breakdown.

***

## 5 · FAQ [#5--faq]

<Accordions type="single">
  <Accordion title="Which models are supported?">
    See the [Models page](https://llmgateway.io/models).
  </Accordion>

  <Accordion title="What makes your service different from OpenRouter?">
    <p>
      Unlike OpenRouter, we offer:
    </p>

    <ul className="list-disc pl-6 mt-2 space-y-1">
      <li>
        Full self-hosting capabilities, giving you complete control over your
        infrastructure
      </li>

      <li>
        Enhanced analytics with deeper insights into your model usage and
        performance
      </li>

      <li>
        No fees when using your own provider keys, maximizing cost efficiency
      </li>

      <li>
        Greater flexibility and customization options for enterprise deployments
      </li>
    </ul>
  </Accordion>

  <Accordion title="How much do you charge for your services?">
    Our pricing structure is designed to be flexible and cost-effective: See the
    [Pricing section](https://llmgateway.io#pricing).
  </Accordion>
</Accordions>

***

## 6 · Next steps [#6--next-steps]

* Read [Self host docs](https://docs.llmgateway.io/self-host) guide.
* Drop into our [GitHub](https://github.com/theopenco/llmgateway) for help or feature requests.

Happy building! ✨


# Health check
URL: https://docs.llmgateway.io/health


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create speech
URL: https://docs.llmgateway.io/v1_audio_speech


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create transcription
URL: https://docs.llmgateway.io/v1_audio_transcriptions


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Chat Completions
URL: https://docs.llmgateway.io/v1_chat_completions


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Embeddings
URL: https://docs.llmgateway.io/v1_embeddings


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Edit image
URL: https://docs.llmgateway.io/v1_images_edits


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create image
URL: https://docs.llmgateway.io/v1_images_generations


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Retrieve key status
URL: https://docs.llmgateway.io/v1_key_retrieve


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Anthropic Messages
URL: https://docs.llmgateway.io/v1_messages


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Models
URL: https://docs.llmgateway.io/v1_models


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Moderations
URL: https://docs.llmgateway.io/v1_moderations


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# OCR
URL: https://docs.llmgateway.io/v1_ocr


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Rerank
URL: https://docs.llmgateway.io/v1_rerank


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Video content
URL: https://docs.llmgateway.io/v1_videos_content


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create video
URL: https://docs.llmgateway.io/v1_videos_create


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Video log content
URL: https://docs.llmgateway.io/v1_videos_log_content


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Retrieve video
URL: https://docs.llmgateway.io/v1_videos_retrieve


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# AI SDK Gateway protocol
URL: https://docs.llmgateway.io/developers/ai-sdk-gateway-protocol


When you pass a bare model string to the AI SDK — `streamText({ model: "anthropic/claude-sonnet-5" })` — the SDK resolves it through its **default provider**, `@ai-sdk/gateway`. That provider does not speak the OpenAI Chat Completions format: it has its own wire protocol, `LanguageModelV*CallOptions` in and `LanguageModelV*` parts out.

LLM Gateway implements that protocol, so an app written against the Vercel AI Gateway runs here with **no code change** — only its base URL and API key are repointed.

## Repoint the default provider [#repoint-the-default-provider]

```ts
import { createGateway } from "@ai-sdk/gateway";

globalThis.AI_SDK_DEFAULT_PROVIDER = createGateway({
	baseURL: "https://api.llmgateway.io/v4/ai",
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});
```

Put this wherever your app runs before its first model call — a Next.js [`instrumentation.ts`](https://nextjs.org/docs/app/guides/instrumentation), a server entrypoint, or a platform-injected preamble. Every bare model string in the app then resolves through LLM Gateway.

`@ai-sdk/gateway` is already a transitive dependency of `ai`, so there is nothing extra to install.

<Callout type="info">
  `@ai-sdk/gateway` reads its API key from `AI_GATEWAY_API_KEY` but has **no**
  environment variable for the base URL — it is a constructor option only. That
  is why repointing takes this one line rather than an env var.
</Callout>

Or construct the provider explicitly and pass it per call:

```ts
import { createGateway } from "@ai-sdk/gateway";
import { streamText } from "ai";

const gateway = createGateway({
	baseURL: "https://api.llmgateway.io/v4/ai",
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const result = streamText({
	model: gateway("anthropic/claude-sonnet-5"),
	prompt: "Hello!",
});
```

## Base URL per AI SDK version [#base-url-per-ai-sdk-version]

The protocol carries the language-model specification version in a request header, and every prefix below serves the same surface — pick the one matching the `@ai-sdk/gateway` your app has, so the default path stays intact:

| AI SDK | Spec version | Base URL                          |
| ------ | ------------ | --------------------------------- |
| 5      | 2            | `https://api.llmgateway.io/v1/ai` |
| 6      | 3            | `https://api.llmgateway.io/v3/ai` |
| 7      | 4            | `https://api.llmgateway.io/v4/ai` |

## Model IDs [#model-ids]

Model IDs use the provider-pinned `provider/model` form (`anthropic/claude-sonnet-5`, `openai/gpt-4o`) — the same convention AI Gateway IDs use, so existing model strings resolve unchanged.

LLM Gateway's smart-routing IDs work here too: pass a bare model ID (`gpt-4o`) to let the gateway pick a provider, or `auto` to let it pick the model. Those are not returned by `getAvailableModels()` because they do not name one provider, but they are accepted.

## Listing models [#listing-models]

```ts
const { models } = await gateway.getAvailableModels();
```

Returns one entry per active provider mapping, with pricing and an AI SDK `specification` block. This is what backs a model picker built on `GatewayModel[]`.

## Credits [#credits]

```ts
const { balance, totalUsed } = await gateway.getCredits();
```

`balance` is the organization's remaining credit balance and `totalUsed` its lifetime credits spend.

## Gateway-only options [#gateway-only-options]

Features that have no field in the AI SDK call options — reasoning effort, service tier, routing strategy, prompt cache keys, plugins — are set through the `llmgateway` provider options namespace, which is passed onto the underlying request:

```ts
const result = streamText({
	model: gateway("openai/gpt-5.6-terra"),
	prompt: "Hello!",
	providerOptions: {
		llmgateway: {
			reasoning_effort: "high",
			routing: "price",
		},
	},
});
```

Any field the [chat completions API](https://docs.llmgateway.io/api-reference) accepts works here, except `model`, `messages` and `stream`, which this surface owns.

## Web search [#web-search]

The provider-native web search tools serialize to provider-defined tools, and the gateway maps them onto its [native web search](https://docs.llmgateway.io/features/web-search):

```ts
import { openai } from "@ai-sdk/openai";

const result = streamText({
	model: gateway("openai/gpt-4o"),
	prompt: "What happened in the news today?",
	tools: { web_search: openai.tools.webSearch() },
});
```

Recognised tools: `openai.web_search`, `openai.web_search_preview`, `anthropic.web_search_20250305`, `anthropic.web_search_20260209`, `google.google_search`. Search results come back as `source-url` message parts plus a provider-executed tool call under the name you bound the tool to, so the AI SDK's sources UI works unchanged.

A provider-defined tool the gateway cannot map is reported as an `unsupported-tool` warning on the result rather than failing the request.

## What is not covered [#what-is-not-covered]

This surface implements language models. Embeddings, images, video, speech, transcription, reranking and realtime are served by the [OpenAI-compatible endpoints](https://docs.llmgateway.io/api-reference) — use [`@llmgateway/ai-sdk-provider`](https://docs.llmgateway.io/developers/ai-sdk) for those.

These call options have no chat completions equivalent and are reported as `unsupported` warnings: `stopSequences`, `seed`, `topK`.


# Image Generation with the AI SDK
URL: https://docs.llmgateway.io/developers/ai-sdk-images


The `@llmgateway/ai-sdk-provider` package supports image generation both through the AI SDK's dedicated `generateImage` function and through chat-based image models that stream images as part of a conversation.

## generateImage [#generateimage]

Use `llmgateway.image()` to get an image model:

```typescript
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { generateImage } from "ai";
import { writeFileSync } from "fs";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const result = await generateImage({
	model: llmgateway.image("gemini-3-pro-image"),
	prompt:
		"A cozy cabin in a snowy mountain landscape at night with aurora borealis",
	size: "1024x1024",
	n: 1,
	// aspectRatio and quality are model-specific — only some providers honor them.
	// aspectRatio works on Gemini image models; OpenAI gpt-image-2 ignores it
	// (use a literal WxH `size` instead).
	aspectRatio: "16:9",
	// quality works on OpenAI gpt-image-2 ("low" | "medium" | "high" | "auto").
	// The AI SDK only forwards it through providerOptions.
	providerOptions: {
		llmgateway: { quality: "high" },
	},
});

result.images.forEach((image, i) => {
	const buf = Buffer.from(image.base64, "base64");
	writeFileSync(`image-${i}.png`, buf);
});
```

<Callout>
  Which sizes, aspect ratios, and quality settings a model accepts depends on
  the model — see [Image Generation](https://docs.llmgateway.io/features/image-generation) for the full
  parameter reference and per-model behavior.
</Callout>

## Chat-based image models [#chat-based-image-models]

Multimodal models like `gemini-3-pro-image` can return images inside a chat conversation. Use `llmgateway.chat()` with `streamText` in a route handler:

```typescript
// app/api/chat/route.ts
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { convertToModelMessages, streamText } from "ai";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = streamText({
		model: llmgateway.chat("gemini-3-pro-image"),
		messages: convertToModelMessages(messages),
	});

	return result.toUIMessageStreamResponse();
}
```

On the client, image parts arrive as message file parts that you can render with the AI Elements `Image` component or a plain `<img>` tag with a data URL. See [Image Generation](https://docs.llmgateway.io/features/image-generation) for the complete `useChat` frontend example.

## Video and audio [#video-and-audio]

The AI SDK does not yet cover the gateway's video and speech endpoints — call them over REST instead:

* [Video Generation](https://docs.llmgateway.io/features/video-generation) — `POST /v1/videos` (async jobs with optional signed callbacks)
* [Speech Generation](https://docs.llmgateway.io/features/speech-generation) — `POST /v1/audio/speech`
* [Transcription](https://docs.llmgateway.io/features/transcription) — `POST /v1/audio/transcriptions`


# Using the AI SDK
URL: https://docs.llmgateway.io/developers/ai-sdk


LLM Gateway ships a first-party provider for the [Vercel AI SDK](https://ai-sdk.dev): [`@llmgateway/ai-sdk-provider`](https://github.com/theopenco/llmgateway-ai-sdk-provider). One provider instance and one API key reach every model in the catalog.

## Install [#install]

```bash
pnpm add ai @llmgateway/ai-sdk-provider
```

Set your API key (create one from the [dashboard](https://llmgateway.io/dashboard)):

```bash
export LLM_GATEWAY_API_KEY=llmgtwy_your_key_here
```

## Generate text [#generate-text]

```typescript
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { generateText } from "ai";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { text } = await generateText({
	model: llmgateway("openai/gpt-4o"),
	prompt: "Hello!",
});
```

Switching models is a one-line change — the same provider serves every model:

```typescript
const { text } = await generateText({
	model: llmgateway("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Hello!",
});
```

## Model ID formats [#model-id-formats]

LLM Gateway supports two model ID formats:

* **Canonical model IDs** (`gpt-4o`) — smart routing picks the best provider based on uptime, throughput, price, and latency
* **Provider-prefixed IDs** (`openai/gpt-4o`) — routes to a specific provider with automatic failover if uptime drops below 90%

See the [routing documentation](https://docs.llmgateway.io/features/routing) for details and the [models page](https://llmgateway.io/models) for the full catalog.

## Stream responses [#stream-responses]

```typescript
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { streamText } from "ai";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { textStream } = await streamText({
	model: llmgateway("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Write a poem about coding",
});

for await (const text of textStream) {
	process.stdout.write(text);
}
```

## Next.js route handler [#nextjs-route-handler]

```typescript
// app/api/chat/route.ts
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { streamText } from "ai";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = await streamText({
		model: llmgateway("openai/gpt-4o"),
		messages,
	});

	return result.toDataStreamResponse();
}
```

## Tool calling [#tool-calling]

```typescript
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { generateText, tool } from "ai";
import { z } from "zod";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { text, toolResults } = await generateText({
	model: llmgateway("openai/gpt-4o"),
	tools: {
		weather: tool({
			description: "Get the weather for a location",
			parameters: z.object({
				location: z.string(),
			}),
			execute: async ({ location }) => {
				return { temperature: 72, condition: "sunny" };
			},
		}),
	},
	prompt: "What's the weather in San Francisco?",
});
```

## Without the provider package [#without-the-provider-package]

If you prefer not to add a dependency, point `@ai-sdk/openai` at the gateway with a custom base URL:

```typescript
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const llmgateway = createOpenAI({
	baseURL: "https://api.llmgateway.io/v1",
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { text } = await generateText({
	model: llmgateway("openai/gpt-4o"),
	prompt: "Hello!",
});
```

<Callout>
  Every request made through the AI SDK shows up in your
  [Activity](https://docs.llmgateway.io/learn/activity) and [Usage & Metrics](https://docs.llmgateway.io/learn/usage-metrics)
  dashboards like any other gateway request — with per-request cost, tokens, and
  latency.
</Callout>

## Generate video [#generate-video]

Version 4 of the provider (AI SDK 7, Node.js 22 or later, ESM) adds `llmgateway.video()` for the SDK's experimental video API. The provider submits a gateway video job, the SDK polls it, and the finished file is downloaded from the authenticated content endpoint.

```typescript
import { writeFile } from "node:fs/promises";
import { llmgateway } from "@llmgateway/ai-sdk-provider";
import { experimental_generateVideo as generateVideo } from "ai";

const { video } = await generateVideo({
	model: llmgateway.video("seedance-2-0"),
	prompt: "A cinematic aerial view of ocean waves at sunrise",
	duration: 8,
	resolution: "1280x720",
	poll: { intervalMs: 5_000, timeoutMs: 600_000 },
});

await writeFile("video.mp4", video.uint8Array);
```

`duration` and a text prompt are required. `resolution` maps to the gateway's `size`, `duration` to `seconds`, and `generateAudio` to `audio`. Pass a first frame as `prompt.image`, first and last frames with `frameImages`, and reference inputs with `inputReferences`; any other gateway field, such as `callback_url`, goes through `providerOptions.llmgateway`. `experimental_startVideo` and `experimental_getVideoStatus` let another process pick up a running job. Supported sizes, durations, and inputs depend on the model — see [video generation](https://docs.llmgateway.io/features/video-generation) for the REST reference. Projects on AI SDK 6 should stay on `@llmgateway/ai-sdk-provider@3`, which has no video support.

## Next steps [#next-steps]

* [Image generation with the AI SDK](https://docs.llmgateway.io/developers/ai-sdk-images)
* [Migrate from Vercel AI Gateway](https://docs.llmgateway.io/migrations/vercel-ai-gateway)
* [Reasoning support](https://docs.llmgateway.io/features/reasoning) and [caching](https://docs.llmgateway.io/features/caching)


# LLM Gateway CLI
URL: https://docs.llmgateway.io/developers/cli


The **LLM Gateway CLI** (`@llmgateway/cli`) is a command-line utility for launching coding agents pre-configured with LLM Gateway, scaffolding projects, discovering models, and managing your LLM Gateway account — API keys, spending budgets, and usage analytics — straight from the terminal.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Installation [#installation]

<Tabs items="[&#x22;npx (Recommended)&#x22;, &#x22;Global Install&#x22;]">
  <Tab value="npx (Recommended)">
    Run commands directly without installation:

    ```bash
    npx @llmgateway/cli init
    ```
  </Tab>

  <Tab value="Global Install">
    Install globally for faster access:

    ```bash
    npm install -g @llmgateway/cli
    ```

    Then run commands directly (`lg` works as a shorthand alias):

    ```bash
    llmgateway init
    lg init
    ```
  </Tab>
</Tabs>

## Quick Start [#quick-start]

<Steps>
  <Step>
    ### Initialize a Project [#initialize-a-project]

    Create a new project from a template:

    ```bash
    npx @llmgateway/cli init
    ```

    Or specify the template and name directly:

    ```bash
    npx @llmgateway/cli init --template image-generation --name my-ai-app
    ```
  </Step>

  <Step>
    ### Sign In [#sign-in]

    Sign in with your LLM Gateway account to unlock key management, budgets, and usage analytics:

    ```bash
    npx @llmgateway/cli auth login --email you@example.com
    ```

    Or store a gateway API key only (enough for making gateway requests):

    ```bash
    npx @llmgateway/cli auth login --key
    ```

    Credentials are stored in `~/.llmgateway/config.json`. The `LLMGATEWAY_API_KEY` environment variable takes precedence over a stored key.
  </Step>

  <Step>
    ### Start Development [#start-development]

    Navigate to your project and start the development server:

    ```bash
    cd my-ai-app
    npx @llmgateway/cli dev
    ```

    Or specify a custom port:

    ```bash
    npx @llmgateway/cli dev --port 3000
    ```
  </Step>
</Steps>

## Launch Coding Agents [#launch-coding-agents]

### `launch` [#launch]

Start any supported coding agent pre-wired to LLM Gateway: one API key, 200+ models, and every request tracked in your [dashboard](https://llmgateway.io/dashboard).

```bash
# Interactive agent picker
npx @llmgateway/cli launch

# Launch a specific agent (shortcuts work too: `llmgateway claude`)
npx @llmgateway/cli launch claude
npx @llmgateway/cli launch opencode
npx @llmgateway/cli launch codex

# Pick a model — launcher flags go before the agent name
npx @llmgateway/cli launch -m gpt-5.5 claude

# Everything after the agent name is passed to the agent itself
npx @llmgateway/cli launch claude --continue

# List all supported agents and see which are installed
npx @llmgateway/cli launch --list

# Inspect what would run without launching
npx @llmgateway/cli launch --dry-run codex
```

Every supported agent also works as a direct shortcut, e.g. `npx @llmgateway/cli claude` or `npx @llmgateway/cli devpass-code`. The launcher configures each agent automatically — environment variables, config files, or the agent's own key-registration command, whichever that agent needs — without overwriting your existing setup. For OpenCode and Claude Code, launching also applies the same model-catalog setup as [`configure`](#configure) on every launch, so their model pickers stay fresh as new models ship.

The API key is resolved from `--key`, the `LLMGATEWAY_API_KEY` environment variable, or the key stored by `llmgateway auth login --key` — in that order. Before launching, the key is verified against the gateway; a stale key (e.g. one you rolled or deleted) is reported with its exact source and the launcher falls back to the next valid one, prompting you for a fresh key if none works. If an agent isn't installed, the launcher prints its official install command and exits.

<Callout type="info">
  See the [integration guides](https://llmgateway.io/guides) for per-agent setup
  details, and run `npx @llmgateway/cli launch --list` for the up-to-date list
  of supported agents.
</Callout>

### `configure` [#configure]

Put LLM Gateway's coding-model catalog directly into an agent's own config, so its model picker lists gateway models without launching through the CLI:

```bash
# opencode: adds every coding model, pinned per upstream provider, to the picker
npx @llmgateway/cli configure opencode

# Claude Code: routes it through LLM Gateway and fills /model from the gateway catalog
npx @llmgateway/cli configure claude

# ...for the current repo only (.claude/settings.local.json)
npx @llmgateway/cli configure claude --project

# Preview without writing
npx @llmgateway/cli configure opencode --dry-run
```

* **OpenCode** — merges `provider/model` entries (e.g. `anthropic/claude-sonnet-5`, `aws-bedrock/claude-sonnet-5`) into `provider.llmgateway.models` in `~/.config/opencode/opencode.json`, with display names, context limits, and per-provider pricing. They show up in the picker as `llmgateway/<provider>/<model>` and pin that upstream provider via the gateway's [provider-routing syntax](https://docs.llmgateway.io/features/routing#provider-specific-routing). OpenCode now ships both catalogs natively too — canonical IDs under &#x2A;*DevPass (LLM Gateway)** (`llmgateway`) and pinned `provider/model` IDs under **LLM Gateway** (`llmgateway-providers`) — so `configure` mainly covers models the built-in catalog has not picked up yet. Existing custom entries and the rest of the file are preserved, and a hand-written `opencode.jsonc` keeps working alongside it.
* **Claude Code** — sets `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, and `CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1` in `~/.claude/settings.json` (requires Claude Code v2.1.129+). Claude Code then loads the gateway's `/v1/models` catalog into its `/model` picker (shown as "From gateway"). Claude Code only lists IDs starting with `claude`/`anthropic`; any other gateway model still works via `claude --model <id>`.

`llmgateway launch opencode` and `llmgateway launch claude` apply the same setup automatically on every launch, keeping the catalog fresh as new models ship.

## Project Commands [#project-commands]

### `init` [#init]

Initialize a new project from a template.

```bash
npx @llmgateway/cli init [directory] [options]
```

**Options:**

* `-t, --template <name>` — Template to use (default: `image-generation`)
* `-n, --name <name>` — Project name

**Examples:**

```bash
# Interactive mode
npx @llmgateway/cli init

# With options
npx @llmgateway/cli init --template image-generation --name my-app
```

### `list` [#list]

Display available project templates, grouped by category. Alias: `ls`.

```bash
npx @llmgateway/cli list
```

**Options:**

* `--json` — Output in JSON format

### `models` [#models]

Browse and filter available AI models.

```bash
npx @llmgateway/cli models [options]
```

**Options:**

* `-c, --capability <type>` — Filter by capability (e.g., `image`, `text`)
* `-p, --provider <name>` — Filter by provider (e.g., `openai`, `anthropic`)
* `-s, --search <term>` — Search models by name
* `--json` — Output in JSON format

**Examples:**

```bash
# List all models
npx @llmgateway/cli models

# Filter by provider
npx @llmgateway/cli models --provider openai

# Search models
npx @llmgateway/cli models --search gpt
```

### `add` [#add]

Add tools or API routes to an existing project.

```bash
npx @llmgateway/cli add [type] [name]
```

Runs interactively when `type` (`tool` or `route`) and `name` are omitted.

**Tools available:**

* `weather` — Weather lookup functionality
* `search` — Web search capability
* `calculator` — Mathematical operations

**API routes available:**

* `generate` — Text generation endpoint
* `chat` — Chat completion endpoint with streaming

### `dev` [#dev]

Start the local development server using your project's package manager.

```bash
npx @llmgateway/cli dev [options]
```

**Options:**

* `-p, --port <number>` — Port to run on

### `upgrade` [#upgrade]

Update LLM Gateway dependencies (`@llmgateway/ai-sdk-provider`, `@llmgateway/models`, `@llmgateway/cli`) in your project.

```bash
npx @llmgateway/cli upgrade [options]
```

**Options:**

* `--check` — Check for updates without installing

### `docs` [#docs]

Open the documentation in your browser.

```bash
npx @llmgateway/cli docs [topic]
```

**Topics:** `models`, `api`, `sdk`, `quickstart` — omit to open the docs home and see all topics.

## Account Commands [#account-commands]

<Callout type="info">
  The commands below require a dashboard session — sign in first with
  `llmgateway auth login --email`. A gateway API key alone is not enough for
  account management.
</Callout>

### `auth` [#auth]

Manage authentication (dashboard session and gateway API key).

```bash
# Sign in with email & password (full access), or paste an API key
npx @llmgateway/cli auth login
npx @llmgateway/cli auth login --email you@example.com
npx @llmgateway/cli auth login --key

# Check authentication status (session + API key)
npx @llmgateway/cli auth status

# Show the signed-in user
npx @llmgateway/cli auth whoami

# Remove stored session and API key
npx @llmgateway/cli auth logout
```

### `keys` [#keys]

Create and manage gateway API keys.

```bash
npx @llmgateway/cli keys <subcommand>
```

#### `keys create` [#keys-create]

Create a new API key, optionally with spending limits and an expiry.

```bash
npx @llmgateway/cli keys create --description "CI key" --limit 100 --expires 30d
```

**Options:**

* `-p, --project <id>` — Project the key belongs to
* `-d, --description <text>` — Key description
* `-l, --limit <usd>` — Total spending limit in USD (e.g. `100` or `49.99`)
* `--period-limit <usd>` — Spending limit per rolling period in USD
* `--period <duration>` — Rolling period for `--period-limit` (`12h`, `1d`, `2w`, `1mo`; default `1mo`)
* `-e, --expires <ttl>` — TTL as a duration (`30d`, `12h`) or an ISO date
* `--json` — Output in JSON format

The token is only displayed once at creation time — save it immediately.

#### `keys list` [#keys-list]

List API keys with spend, budget, and expiry. Alias: `keys ls`.

**Options:**

* `-p, --project <id>` — Filter by project
* `--all` — Show all keys in the org (admin/owner only)
* `--json` — Output in JSON format

#### `keys update <id>` [#keys-update-id]

Activate or deactivate an API key.

**Options:**

* `--activate` — Set the key to active
* `--deactivate` — Set the key to inactive
* `-e, --expires <ttl>` — New expiry as a duration (`30d`) or ISO date (needed to reactivate expired keys)

#### `keys limit <id>` [#keys-limit-id]

Set spending limits on an API key (same as `budget set`).

**Options:**

* `-l, --limit <usd>` — Total spending limit in USD
* `--period-limit <usd>` — Spending limit per rolling period in USD
* `--period <duration>` — Rolling period (`12h`, `1d`, `2w`, `1mo`; default `1mo`)
* `--clear` — Remove all spending limits

#### `keys roll <id>` [#keys-roll-id]

Regenerate the token for an API key. The old token becomes invalid immediately.

**Options:**

* `-y, --yes` — Skip confirmation

#### `keys delete <id>` [#keys-delete-id]

Delete an API key. Alias: `keys rm`.

**Options:**

* `-y, --yes` — Skip confirmation

### `budget` [#budget]

Manage API key spending limits.

```bash
# Set a total and/or rolling-period budget
npx @llmgateway/cli budget set <keyId> --limit 100 --period-limit 25 --period 1w

# Remove all spending limits
npx @llmgateway/cli budget set <keyId> --clear

# Show budget and current spend
npx @llmgateway/cli budget get <keyId>
```

**`budget set` options:** `-l, --limit <usd>`, `--period-limit <usd>`, `--period <duration>`, `--clear`

**`budget get` options:** `-p, --project <id>`, `--json`

### `usage` [#usage]

View usage and cost analytics.

```bash
npx @llmgateway/cli usage [options]
```

**Options:**

* `-o, --org <id>` — Aggregate usage across an organization
* `-p, --project <id>` — Filter by project
* `-k, --api-key <id>` — Filter by API key
* `--by <dimension>` — Break down by `model` or `key`
* `-r, --range <range>` — Time range: `1h`, `4h`, `24h`, `7d`, `30d`, `365d` (default `7d`)
* `--days <n>` — Look back N days instead of `--range`
* `--from <date>` / `--to <date>` — Custom date range (`YYYY-MM-DD`)
* `--json` — Output in JSON format

**Examples:**

```bash
# Last 7 days for the default project
npx @llmgateway/cli usage

# Cost per model over the last 30 days
npx @llmgateway/cli usage --by model --range 30d

# Whole-org aggregate
npx @llmgateway/cli usage --org <orgId>
```

#### `usage sources` [#usage-sources]

Break down usage by session/agent source to see which agents or sessions are spending.

```bash
npx @llmgateway/cli usage sources [options]
```

**Options:** `-p, --project <id>`, `-r, --range <range>` (`7d`, `30d`), `--from <date>`, `--to <date>`, `--json`

### `orgs` [#orgs]

List your organizations with plan and credit balance. Alias: `orgs ls`.

```bash
npx @llmgateway/cli orgs list [--json]
```

### `projects` [#projects]

Manage projects and the CLI's default project.

```bash
# List projects (optionally filtered by org)
npx @llmgateway/cli projects list [--org <id>] [--json]

# Set the default project used by keys/budget/usage commands
npx @llmgateway/cli projects use <id>
```

### `credits` [#credits]

Show organization credit balances.

```bash
npx @llmgateway/cli credits [--org <id>] [--json]
```

## Available Templates [#available-templates]

### Web Applications [#web-applications]

* **`embeddable-credits`** — Monetize your AI app in 5 minutes. End-user wallets and in-app credit purchases ("Stripe for AI") with the embeddable SDK.
* **`image-generation`** — Full-stack AI image generation app (Next.js 16, React 19). Multi-provider support with a unified API.
* **`ai-chatbot`** — AI chatbot with streaming responses.
* **`og-image-generator`** — AI-powered OG image generator.
* **`feedback-dashboard`** — Customer feedback sentiment dashboard.
* **`writing-assistant`** — AI writing assistant with text actions.
* **`qa-agent`** — AI-powered QA testing agent with browser automation, real-time action timeline, and live browser preview.
* **`showcase`** — Public, filterable gallery of apps built with LLM Gateway templates. Static and deployable, with a "Submit your app" flow.

### Bots [#bots]

* **`slack-qa-bot`** — Slack bot that streams AI answers and keeps thread context.

### CLI Agents [#cli-agents]

* **`weather-agent`** — Answers weather queries using tool calling.
* **`lead-agent`** — Researches people and posts results to Discord.
* **`changelog-generator-agent`** — Generates changelogs from git history.
* **`email-drafter-agent`** — Drafts polished emails from rough notes.
* **`sentiment-analyzer-agent`** — Analyzes text sentiment.
* **`data-extractor-agent`** — Extracts structured entities from text.

```bash
npx @llmgateway/cli init --template qa-agent
```

## Configuration [#configuration]

The CLI stores configuration in `~/.llmgateway/config.json`:

```json
{
	"apiKey": "llmgtwy_...",
	"defaultTemplate": "image-generation",
	"sessionEmail": "you@example.com",
	"defaultOrgId": "org_...",
	"defaultProjectId": "proj_..."
}
```

Signing in with `auth login --email` also stores a dashboard session used by the account commands (`keys`, `budget`, `usage`, `orgs`, `projects`, `credits`).

### Environment Variables [#environment-variables]

* `LLMGATEWAY_API_KEY` — Gateway API key; takes precedence over the config file:

```bash
export LLMGATEWAY_API_KEY="llmgtwy_..."
```

* `LLMGATEWAY_API_URL` — Override the management API base URL (defaults to `https://internal.llmgateway.io`), useful for self-hosted deployments.

## More Resources [#more-resources]

* [Agents](https://llmgateway.io/agents) — Pre-built AI agents
* [Templates](https://llmgateway.io/templates) — Production-ready starter projects
* [GitHub Repository](https://github.com/theopenco/llmgateway-templates) — Source code and issues

<Callout type="info">
  Need help or want to request a feature? Open an issue on
  [GitHub](https://github.com/theopenco/llmgateway-templates/issues).
</Callout>


# DevPass Usage API
URL: https://docs.llmgateway.io/developers/devpass-usage


Use `GET /v1/key` to show DevPass allowance meters in a coding tool or other
trusted application. The endpoint uses the application's existing gateway API
key, so it does not require a dashboard session.

<Callout type="warning">
  A regular gateway API key is a secret. Do not embed it in public browser code
  or distribute it with an application. Read it from the user's secure local
  configuration or call the endpoint from a trusted backend.
</Callout>

## Request [#request]

```bash
curl https://api.llmgateway.io/v1/key \
  -H "Authorization: Bearer llmgtwy_your_api_key_here"
```

Only regular gateway API keys can use this endpoint. Platform publishable keys
and end-user sessions receive `403 Forbidden` because they cannot read
organization-level plan state.

## Response [#response]

```json
{
	"data": {
		"label": "My coding tool",
		"usage": "31.42",
		"limit": null,
		"devPlan": "pro",
		"devPlanCreditsUsed": "25",
		"devPlanCreditsLimit": "237",
		"devPlanCreditsRemaining": "212.00",
		"devPlanPremiumWeeklyLimit": "35.55",
		"devPlanPremiumCreditsUsed": "5.00",
		"devPlanPremiumWeekResetsAt": "2026-08-28T12:00:00.000Z"
	}
}
```

| Field                        | Meaning                                                        |
| ---------------------------- | -------------------------------------------------------------- |
| `label`                      | Description of the key                                         |
| `usage`                      | All-time usage attributed to the key, in USD                   |
| `limit`                      | All-time key usage limit in USD, or `null`                     |
| `devPlan`                    | `lite`, `pro`, `max`, or `none`                                |
| `devPlanCreditsUsed`         | Plan credits used in the current billing cycle                 |
| `devPlanCreditsLimit`        | Plan credit allowance for the current billing cycle            |
| `devPlanCreditsRemaining`    | Remaining plan credits, clamped to zero                        |
| `devPlanPremiumWeeklyLimit`  | Weekly premium-model allowance                                 |
| `devPlanPremiumCreditsUsed`  | Premium-model credits used in the current weekly window        |
| `devPlanPremiumWeekResetsAt` | ISO 8601 reset time, or `null` when no weekly window is active |

All USD values are decimal strings. Parse them before calculating percentages
or remaining premium allowance.

## Weekly window behavior [#weekly-window-behavior]

The premium window starts with the first premium-model request and lasts seven
days. When it expires, the endpoint returns `"0.00"` for
`devPlanPremiumCreditsUsed` and `null` for `devPlanPremiumWeekResetsAt`; the
full weekly allowance is already available, and the next premium request starts
a new window.

Pay-as-you-go keys return `"none"` for `devPlan` and zero for every DevPass
field. This lets one integration support both key types without a separate
account lookup.

The endpoint remains readable when the key has exceeded its own usage limit,
so an integration can explain why requests stopped. Invalid or inactive keys
receive `401 Unauthorized`.

See [Retrieve key status](https://docs.llmgateway.io/v1_key_retrieve) for the generated API reference and
complete response schema.


# Overview
URL: https://docs.llmgateway.io/developers


This section is for developers building applications on top of LLM Gateway — with our command-line tool, the MCP server, the [Vercel AI SDK](https://ai-sdk.dev) via our first-party provider package [`@llmgateway/ai-sdk-provider`](https://github.com/theopenco/llmgateway-ai-sdk-provider), and [TanStack AI](https://tanstack.com/ai) via the first-party [`@tanstack/ai-llmgateway`](https://www.npmjs.com/package/@tanstack/ai-llmgateway) adapter.

## Guides [#guides]

* [**DevPass Usage API**](https://docs.llmgateway.io/developers/devpass-usage) — Show monthly and weekly DevPass allowance meters in your application
* [**LLM Gateway CLI**](https://docs.llmgateway.io/developers/cli) — Launch coding agents, scaffold projects from templates, generate agent configs, and manage keys, budgets, and usage from the terminal
* [**Model Context Protocol (MCP)**](https://docs.llmgateway.io/developers/mcp) — Use LLM Gateway as an MCP server from Claude Code, Cursor, and other MCP clients
* [**Using the AI SDK**](https://docs.llmgateway.io/developers/ai-sdk) — Install the provider, generate and stream text, call tools, and wire up Next.js routes
* [**Image Generation with the AI SDK**](https://docs.llmgateway.io/developers/ai-sdk-images) — Generate images with `generateImage` and stream image output through chat
* [**Using TanStack AI**](https://docs.llmgateway.io/developers/tanstack-ai) — Stream chat with `useChat`, call tools, and surface reasoning through the first-party `@tanstack/ai-llmgateway` adapter

## Why the AI SDK [#why-the-ai-sdk]

The AI SDK gives you one TypeScript interface for text generation, streaming, tool calling, and image generation. Combined with LLM Gateway, a single provider instance and one API key reach every model in the catalog — see the [models page](https://llmgateway.io/models) for what's available.

## Other ways to integrate [#other-ways-to-integrate]

If you're not using the AI SDK:

* Use any OpenAI-compatible SDK against `https://api.llmgateway.io/v1` — see the [Quickstart](https://docs.llmgateway.io/quick-start)
* Use the Anthropic SDK against the [Anthropic-compatible endpoint](https://docs.llmgateway.io/features/anthropic-endpoint)
* Call the REST API directly — see the API reference in the sidebar


# Model Context Protocol (MCP)
URL: https://llmgateway.io/guides/mcp


Connect your AI assistant to LLM Gateway to inspect your usage and costs, discover your most-used models, providers and coding apps, and generate text or images. The same API key connects all of these tools.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`model-id` instead of `provider/model-id`) —
  provider-pinned routing is not available on coding plans; the gateway picks
  the provider for you.
</Callout>

## What is MCP? [#what-is-mcp]

The Model Context Protocol (MCP) is an open standard that allows AI assistants to connect with external tools and data sources. LLM Gateway's MCP server exposes tools for:

* **Account and usage analytics** - Check spending limits, request/token totals, costs, trends, and provider/model/app rankings
* **Chat completions** - Send messages to any supported LLM
* **Image generation** - Generate images using models like Qwen Image
* **Nano Banana image generation** - Generate images with Gemini 3 Pro Image and optionally save to disk
* **Model discovery** - List available models with capabilities and pricing

## Available Tools [#available-tools]

### `get-account` [#get-account]

Inspect the connected user, organization, project, role, analytics scope, and API key spending limits. Owners and admins also receive the organization's current credit balance. The balance is not a DevPass plan allowance. No parameters are required, and credentials are never returned.

### `get-usage` [#get-usage]

Get request/token totals, errors, cache hits, costs, a time series, and your most-used provider, model, and coding agent/app **by request count**.

| Parameter     | Description                                                                        |
| ------------- | ---------------------------------------------------------------------------------- |
| `from`        | Optional first UTC date, `YYYY-MM-DD`, inclusive. Defaults to 29 days before `to`. |
| `to`          | Optional last UTC date, inclusive. Defaults to today. Maximum range: 366 days.     |
| `granularity` | `day` (default) or `hour`. Hourly reports allow at most 31 days.                   |

```json
{
	"from": "2026-08-01",
	"to": "2026-08-31",
	"granularity": "day"
}
```

The response includes `scope`, the resolved dates, `updatedAt`, `totals`, `series`, `mostUsedProvider`, `mostUsedModel`, `mostUsedApp`, and `appUsageCoverage`. Only time buckets with activity appear in `series`. Empty periods return zero totals, an empty series, and null rankings.

### `get-usage-breakdown` [#get-usage-breakdown]

Rank providers, models, coding apps, or API keys by requests, inference cost, or tokens.

| Parameter    | Description                                                             |
| ------------ | ----------------------------------------------------------------------- |
| `group_by`   | Required: `provider`, `model`, `app`, or `api_key`.                     |
| `sort_by`    | `requests` (default), `cost`, or `tokens`, descending. Ties use the ID. |
| `from`, `to` | Same inclusive UTC dates as `get-usage`.                                |
| `limit`      | Results per page: 1–100, default 10.                                    |
| `offset`     | Results to skip: 0–10000, default 0.                                    |

```json
{
	"group_by": "app",
	"sort_by": "cost",
	"limit": 10
}
```

The response includes each row's ID, display name, requests, tokens and costs, plus `pagination.hasMore` and `coverage`. Increase `offset` by `limit` to get the next page. Known app aliases are combined before ranking. `unknown` identifies requests with no recorded source; custom app names remain as recorded.

### Analytics scope and cost fields [#analytics-scope-and-cost-fields]

* Owners and admins see the **connected project's** usage across its keys. Developers see only the keys they created in that project, including inactive-key history.
* A project API key does not grant access to another project or organization. Tools do not accept scope overrides. Use a key for the project you want to inspect.
* Use an active user API key. Customer credentials and expired or revoked keys cannot read account analytics. Project access is checked on every analytics request.
* Analytics tools are read-only, incur no model charges, and remain available when a key or member reaches a spending limit. Generation tools still enforce those limits.
* `costUsd` is inference usage cost. `creditsCostUsd` and `byokCostUsd` separate gateway credits from provider costs paid with your own provider keys. `dataStorageCostUsd` is separate. These are usage statistics, not invoice totals or exact changes in credit balance.
* Statistics come from hourly aggregates, survive request-retention cleanup, and may lag recent requests. `updatedAt` reports the last summary aggregation in the requested period.
* App attribution uses the request's recorded source, including recognized coding clients and `x-source` values. MCP generation calls preserve client attribution headers. Configure `x-source` on your MCP connection if your client does not identify itself. Attribution does not identify which person used a shared key.
* Historical per-key app statistics start when per-key source aggregation is enabled. `appUsageCoverage` / `coverage` compare recorded breakdown requests with total requests; `complete: false` means rankings cover only part of the period. This differs from an `unknown` source, which is a recorded request without app attribution.

All three tools return JSON in both `structuredContent` and a text content block for older clients. An unavailable backend produces a tool error, never a fabricated zero-usage report.

### `chat` [#chat]

Send a message to any LLM and get a response.

**Parameters:**

* `model` (string) - A model ID from `list-models` or the [live catalog](https://llmgateway.io/models)
* `messages` (array) - Array of messages with `role` and `content`
* `temperature` (number, optional) - Sampling temperature (0-2)
* `max_tokens` (number, optional) - Maximum tokens to generate

**Example:**

```json
{
	"model": "MODEL_ID",
	"messages": [{ "role": "user", "content": "Explain quantum computing" }],
	"temperature": 0.7
}
```

### `generate-image` [#generate-image]

Generate images from text prompts using AI image models.

**Parameters:**

* `prompt` (string) - Text description of the image to generate
* `model` (string, optional) - Image model (default: `"qwen-image-plus"`)
* `size` (string, optional) - Image size (default: `"1024x1024"`)
* `n` (number, optional) - Number of images (1-4, default: 1)

**Example:**

```json
{
	"prompt": "A serene mountain landscape at sunset",
	"model": "qwen-image-max",
	"size": "1024x1024"
}
```

### `generate-nano-banana` [#generate-nano-banana]

Generate an image using Gemini 3 Pro Image ("Nano Banana Pro"). Returns an inline image preview, and optionally saves the image to disk when the server is configured with an upload directory.

**Parameters:**

* `prompt` (string) - Text description of the image to generate
* `filename` (string, optional) - Filename for the saved image, no path separators allowed (default: `nano-banana-{timestamp}.png`)
* `aspect_ratio` (string, optional) - Aspect ratio: `"1:1"`, `"16:9"`, `"4:3"`, or `"5:4"`

**Example:**

```json
{
	"prompt": "A pixel-art cat sitting on a rainbow",
	"filename": "hero-image.png",
	"aspect_ratio": "16:9"
}
```

<Callout type="info">
  **Saving images to disk** requires the `UPLOAD_DIR` environment variable to be
  set on the MCP server. When set, images are saved to that directory. Without
  it, images are returned inline only — no files are written to disk. See
  [Enabling local image saving](#enabling-local-image-saving) for setup
  instructions.
</Callout>

### `list-models` [#list-models]

List available LLM models with capabilities and pricing.

**Parameters:**

* `include_deactivated` (boolean, optional) - Include deactivated models
* `exclude_deprecated` (boolean, optional) - Exclude deprecated models
* `limit` (number, optional) - Maximum models to return (default: 20)
* `family` (string, optional) - Filter by model family

### `list-image-models` [#list-image-models]

List all available image generation models.

Use the tool for current model IDs, capabilities, and pricing, or browse the [live models page](https://llmgateway.io/models).

## Setup [#setup]

<Tabs items="[&#x22;Claude Code&#x22;, &#x22;Codex&#x22;, &#x22;Cursor&#x22;, &#x22;Other MCP Clients&#x22;]">
  <Tab value="Claude Code">
    <Steps>
      <Step>
        ### Get Your API Key [#get-your-api-key]

        1. Log in to your [LLM Gateway dashboard](https://llmgateway.io/dashboard)
        2. Navigate to **API Keys** section
        3. Create a new API key and copy it
      </Step>

      <Step>
        ### Configure Claude Code [#configure-claude-code]

        Run the following command in your terminal:

        ```bash
        claude mcp add --transport http --scope user llmgateway https://api.llmgateway.io/mcp \
          --header "Authorization: Bearer your-api-key-here"
        ```

        <Callout type="info">
          **Alternative: Manual configuration**

          You can also add the MCP server manually by editing `~/.claude.json` (user scope) or `.mcp.json` in your project root (project scope):

          ```json
          {
            "mcpServers": {
              "llmgateway": {
                "url": "https://api.llmgateway.io/mcp",
                "headers": {
                  "Authorization": "Bearer your-api-key-here"
                }
              }
            }
          }
          ```

          Restart Claude Code after manual configuration changes.
        </Callout>
      </Step>

      <Step>
        ### Test the Integration [#test-the-integration]

        Try using the tools in Claude Code:

        * "Show my usage and costs for the last 30 days"
        * "Generate an image of a futuristic city using the generate-image tool"
        * "Use generate-nano-banana to create a hero image for my landing page"
        * "Which model, provider, and coding app do I use most?"
      </Step>
    </Steps>
  </Tab>

  <Tab value="Codex">
    <Steps>
      <Step>
        ### Get Your API Key [#get-your-api-key-1]

        1. Log in to your [LLM Gateway dashboard](https://llmgateway.io/dashboard)
        2. Navigate to **API Keys** section
        3. Create a new API key and copy it
        4. Set it as an environment variable: `export LLM_GATEWAY_API_KEY="your-api-key-here"`
      </Step>

      <Step>
        ### Configure Codex [#configure-codex]

        Run the following command in your terminal:

        ```bash
        codex mcp add llmgateway --url https://api.llmgateway.io/mcp \
          --bearer-token-env-var LLM_GATEWAY_API_KEY
        ```

        <Callout type="info">
          **Alternative: Manual configuration**

          You can also add the MCP server manually by editing `~/.codex/config.toml`:

          ```toml
          [mcp_servers.llmgateway]
          url = "https://api.llmgateway.io/mcp"
          bearer_token_env_var = "LLM_GATEWAY_API_KEY"
          ```
        </Callout>
      </Step>

      <Step>
        ### Test the Integration [#test-the-integration-1]

        Run `/mcp` in the Codex TUI to confirm the `llmgateway` server is connected. Try:

        * "Show my usage and costs for the last 30 days"
        * "Generate an image of a futuristic city using the generate-image tool"
        * "Use generate-nano-banana to create a hero image for my landing page"
        * "Which model, provider, and coding app do I use most?"
      </Step>
    </Steps>
  </Tab>

  <Tab value="Cursor">
    <Steps>
      <Step>
        ### Get Your API Key [#get-your-api-key-2]

        1. Log in to your [LLM Gateway dashboard](https://llmgateway.io/dashboard)
        2. Navigate to **API Keys** section
        3. Create a new API key and copy it
      </Step>

      <Step>
        ### Configure Cursor [#configure-cursor]

        Add the following to your Cursor MCP configuration file (`~/.cursor/mcp.json`):

        ```json
        {
          "mcpServers": {
            "llmgateway": {
              "url": "https://api.llmgateway.io/mcp",
              "headers": {
                "Authorization": "Bearer your-api-key-here"
              }
            }
          }
        }
        ```

        Or open the Command Palette (`Cmd/Ctrl + Shift + P&#x60;), search for &#x2A;*"Cursor Settings"**, then go to **Tools & Integrations** > **Add Custom MCP** and paste the configuration above.

        <Callout type="info">
          Cursor v0.48.0+ is required for Streamable HTTP MCP support.
        </Callout>
      </Step>

      <Step>
        ### Test the Integration [#test-the-integration-2]

        Open a chat in **Agent Mode**, click the **Select Tools** icon, and verify the LLM Gateway tools appear. Try:

        * "Show my usage and costs for the last 30 days"
        * "Generate an image of a futuristic city using the generate-image tool"
        * "Use generate-nano-banana to create a hero image for my landing page"
        * "Which model, provider, and coding app do I use most?"
      </Step>
    </Steps>
  </Tab>

  <Tab value="Other MCP Clients">
    LLM Gateway's MCP server supports the standard HTTP Streamable transport. Configure your client with:

    * **Endpoint:** `https://api.llmgateway.io/mcp`
    * **Authentication:** Bearer token via `Authorization` header or `x-api-key` header
    * **Protocol Version:** 2024-11-05

    **Direct HTTP Example:**

    ```bash
    curl -X POST https://api.llmgateway.io/mcp \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer your-api-key" \
      -d '{
        "jsonrpc": "2.0",
        "id": 1,
        "method": "tools/list"
      }'
    ```

    **Server-Sent Events (SSE):**

    For real-time updates, connect with `Accept: text/event-stream`:

    ```bash
    curl -N https://api.llmgateway.io/mcp \
      -H "Accept: text/event-stream" \
      -H "Authorization: Bearer your-api-key"
    ```
  </Tab>
</Tabs>

## Use Cases [#use-cases]

### Usage and Spending [#usage-and-spending]

```text
What did I spend this month, and which coding app accounts for the most cost?
Show my most-used model and provider, then compare daily usage with last month.
```

Use `get-account` to confirm the scope, `get-usage` for each period, and `get-usage-breakdown` with `group_by: "app"` and `sort_by: "cost"` for the app ranking.

### Multi-Model Access in Claude Code [#multi-model-access-in-claude-code]

Use Claude Code to interact with models it doesn't natively support:

```
List available models, then use the chat tool with a suitable model to review this code.
```

### Image Generation [#image-generation]

Generate images directly from your AI assistant:

```
Use generate-image to create a logo for my new startup.
It should be minimalist, blue and white, representing AI and cloud computing.
```

### Nano Banana (Gemini Image Generation) [#nano-banana-gemini-image-generation]

Generate images with Gemini 3 Pro for use in your project:

```
Use generate-nano-banana to create a hero image for my landing page with a 16:9 aspect ratio.
```

### Cost-Effective Model Selection [#cost-effective-model-selection]

Query available models to find the best option for your task:

```
List models and their pricing, then choose a suitable low-cost model for this task.
```

## Authentication [#authentication]

The MCP server supports two authentication methods:

1. **Bearer Token** - `Authorization: Bearer your-api-key`
2. **API Key Header** - `x-api-key: your-api-key`

Use the same project API key you use for inference. Analytics access follows the [scope rules above](#analytics-scope-and-cost-fields).

## OAuth Support [#oauth-support]

For applications that prefer OAuth authentication, LLM Gateway's MCP server implements OAuth 2.0:

* **Authorization Endpoint:** `/oauth/authorize`
* **Token Endpoint:** `/oauth/token`
* **Registration Endpoint:** `/oauth/register`
* **Supported Flows:** Authorization Code, Client Credentials

## Enabling Local Image Saving [#enabling-local-image-saving]

By default, `generate-nano-banana` returns images inline without writing to disk. To enable saving generated images to the server filesystem, the `UPLOAD_DIR` environment variable must be set on the **gateway host** at startup. This is a server-side setting — it cannot be configured from the client.

This is only possible for **self-hosted** MCP deployments. Configure `UPLOAD_DIR` using your deployment method:

* **Docker:** Pass `-e UPLOAD_DIR=/data/images` or add it to your `docker-compose.yml` environment section.
* **systemd:** Add `Environment=UPLOAD_DIR=/data/images` to your service unit file.
* **.env file:** Add `UPLOAD_DIR=/data/images` to the `.env` file loaded by your gateway process.

<Callout type="warn">
  The shared hosted endpoint (`api.llmgateway.io`) does not support configuring
  `UPLOAD_DIR`. On the hosted service, images are always returned inline — no
  files are written to disk. To enable server-side image saving, you must
  self-host the MCP server and set `UPLOAD_DIR` at startup.
</Callout>

## Troubleshooting [#troubleshooting]

### Connection Errors [#connection-errors]

If you're having trouble connecting:

1. Verify your API key is valid
2. Check the endpoint URL is correct: `https://api.llmgateway.io/mcp`
3. Ensure your firewall allows outbound HTTPS connections

### Tool Not Found [#tool-not-found]

If tools aren't appearing:

1. Restart your MCP client
2. Check the configuration syntax
3. Verify the MCP server is responding: `GET https://api.llmgateway.io/mcp`

### Rate Limiting [#rate-limiting]

The MCP server respects your account's rate limits. If you're hitting limits:

1. Check your usage in the dashboard
2. Consider upgrading your plan
3. Implement request queuing in your application

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support.
</Callout>

## Benefits [#benefits]

* **Unified Access** - Discover the [live model and provider catalog](https://llmgateway.io/models) through one interface
* **Cost Tracking** - Ask your assistant about usage, spending, and your most-used models, providers, and apps
* **Caching** - Automatic response caching reduces costs and latency
* **Fallback** - Automatic provider failover ensures reliability
* **Image Generation** - Generate images directly from your AI assistant


# Using TanStack AI
URL: https://docs.llmgateway.io/developers/tanstack-ai


[TanStack AI](https://tanstack.com/ai) ships a first-party LLM Gateway adapter: [`@tanstack/ai-llmgateway`](https://www.npmjs.com/package/@tanstack/ai-llmgateway), maintained in the TanStack AI repository alongside the OpenAI and Anthropic adapters. One adapter and one API key reach every model in the catalog.

## Install [#install]

```bash
pnpm add @tanstack/ai @tanstack/ai-react @tanstack/ai-llmgateway
```

`@tanstack/ai-react` is the React client; TanStack AI also ships Vue, Svelte, Angular, and Preact packages that work with the same adapter.

Set your API key (create one from the [dashboard](https://llmgateway.io/dashboard)):

```bash
export LLM_GATEWAY_API_KEY=llmgtwy_your_key_here
```

## Stream chat from a server route [#stream-chat-from-a-server-route]

`llmGatewayText(model)` creates the adapter and reads the key from `LLM_GATEWAY_API_KEY`:

```typescript
// app/api/chat/route.ts
import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

export async function POST(request: Request) {
	const { messages } = await request.json();

	const stream = chat({
		adapter: llmGatewayText("gpt-5.6-terra"),
		messages,
	});

	return toServerSentEventsResponse(stream);
}
```

Switching models is a one-line change — the same adapter serves every model:

```typescript
adapter: llmGatewayText("claude-sonnet-5"),
```

## Model ID formats [#model-id-formats]

LLM Gateway supports two model ID formats:

* **Canonical model IDs** (`gpt-5.6-terra`) — smart routing picks the best provider based on uptime, throughput, price, and latency
* **Provider-prefixed IDs** (`moonshot/kimi-k3`) — routes to a specific provider with automatic failover if uptime drops below 90%

A curated set of flagship models carries typed metadata (`LLMGATEWAY_CHAT_MODELS`) with editor autocomplete for input modalities and options; any other ID from the [models page](https://llmgateway.io/models) still works. See the [routing documentation](https://docs.llmgateway.io/features/routing) for details.

## Connect the React client [#connect-the-react-client]

`useChat` consumes the AG-UI event stream from the route above — no client-side API key, no per-provider wiring:

```tsx
// components/chat.tsx
"use client";

import { fetchServerSentEvents, useChat } from "@tanstack/ai-react";
import { useState } from "react";

export function Chat() {
	const [input, setInput] = useState("");
	const { messages, sendMessage, isLoading } = useChat({
		connection: fetchServerSentEvents("/api/chat"),
	});

	return (
		<div>
			{messages.map((message) => (
				<div key={message.id}>
					<strong>{message.role === "assistant" ? "Assistant" : "You"}</strong>
					{message.parts.map((part, index) =>
						part.type === "text" ? <p key={index}>{part.content}</p> : null,
					)}
				</div>
			))}
			<form
				onSubmit={(event) => {
					event.preventDefault();
					if (!input.trim() || isLoading) {
						return;
					}
					sendMessage(input);
					setInput("");
				}}
			>
				<input
					value={input}
					onChange={(event) => setInput(event.target.value)}
					placeholder="Say something..."
				/>
			</form>
		</div>
	);
}
```

## Tool calling [#tool-calling]

Define tools with `toolDefinition` and attach a server handler — TanStack AI runs the tool loop for you:

```typescript
import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";
import { z } from "zod";

const getWeather = toolDefinition({
	name: "get_weather",
	description: "Get the current weather for a location",
	inputSchema: z.object({
		location: z.string(),
	}),
}).server(async ({ location }) => {
	return { temperature: 72, condition: "sunny" };
});

export async function POST(request: Request) {
	const { messages } = await request.json();

	const stream = chat({
		adapter: llmGatewayText("gpt-5.6-terra"),
		messages,
		tools: [getWeather],
	});

	return toServerSentEventsResponse(stream);
}
```

## Reasoning models [#reasoning-models]

Reasoning models stream their thinking as `reasoning_content` deltas, which the adapter surfaces as AG-UI `REASONING_*` events — they arrive in `useChat` as `thinking` parts. Control the depth with `reasoning_effort` in `modelOptions`:

```typescript
const stream = chat({
	adapter: llmGatewayText("kimi-k3"),
	messages,
	modelOptions: {
		temperature: 0.7,
		reasoning_effort: "high",
	},
});
```

`reasoning_effort` accepts the extended scale `none` / `minimal` / `low` / `medium` / `high` / `xhigh` / `max`; which tiers a model honors depends on the model and the provider it routes to. Parameters a routed provider doesn't support are stripped server-side, so `modelOptions` stay portable across models. See [reasoning support](https://docs.llmgateway.io/features/reasoning).

## Summarization [#summarization]

The adapter also covers TanStack AI's `summarize` surface:

```typescript
import { summarize } from "@tanstack/ai";
import { llmGatewaySummarize } from "@tanstack/ai-llmgateway";

const result = await summarize({
	adapter: llmGatewaySummarize("gpt-5.4-mini"),
	text: "Long article text...",
	stream: false,
});
```

## Self-hosted deployments [#self-hosted-deployments]

`createLLMGatewayText` takes the key explicitly plus a `baseURL` for self-hosted gateways:

```typescript
import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText(
	"gpt-5.6-terra",
	process.env.LLM_GATEWAY_API_KEY!,
	{
		baseURL: "https://gateway.internal.example.com/v1",
	},
);
```

<Callout>
  Every request made through TanStack AI shows up in your
  [Activity](https://docs.llmgateway.io/learn/activity) and [Usage & Metrics](https://docs.llmgateway.io/learn/usage-metrics)
  dashboards like any other gateway request — with per-request cost, tokens, and
  latency.
</Callout>

## Next steps [#next-steps]

* [TanStack AI adapter reference](https://tanstack.com/ai/latest/docs/adapters/llmgateway)
* [Routing and fallback](https://docs.llmgateway.io/features/routing)
* [Reasoning support](https://docs.llmgateway.io/features/reasoning) and [caching](https://docs.llmgateway.io/features/caching)


# Airside for Providers
URL: https://docs.llmgateway.io/features/airside


[Airside](https://airside.llmgateway.io) is the self-serve console for the companies behind the models — the [providers](https://llmgateway.io/providers) LLM Gateway routes to. It runs on an airport metaphor: LLM Gateway is the airport, developers are passengers, and providers are **carriers**. As a carrier you claim your provider, register your fleet of models, file your fares, and watch traffic arrive.

## Claiming your carrier [#claiming-your-carrier]

Sign up at [airside.llmgateway.io](https://airside.llmgateway.io) with your **company email** and verify it. A catalogue provider is claimable when the registrable domain of your verified email matches the registrable domain of the provider's API endpoint (or its website) — if your API is served from `api.example.ai`, an `@example.ai` address can claim it. Subdomains collapse to the registrable domain, so `ops@mail.example.ai` works too.

Every claim is reviewed by the LLM Gateway team before the carrier goes live; a rejected claim shows the review note so you can follow up. One company can operate several carriers — regional deployments or separate brands all live under one console.

Listing on llmgateway.io carries a one-time **$2,500 listing fee** per provider company, paid during onboarding. Providers we already work with receive an **invite code** that waives it — entered in the same onboarding step instead of paying. See the [pricing summary](https://airside.llmgateway.io/pricing.md) for the full economics.

## Crew [#crew]

A listing covers your whole team. Under **Crew**, company owners invite teammates by email — up to **10 members** per company, pending invites included. Invites are limited to your team's domains (the ones that prove your claims); a teammate with an existing account joins instantly, anyone else joins the first time they sign in with the invited address.

## Registering a new carrier [#registering-a-new-carrier]

Not in the [provider catalogue](https://llmgateway.io/providers) yet? Register your provider as a **new carrier** from the onboarding page: pick a carrier id, display name, and the base URL of your **OpenAI-compatible API** (the gateway calls `<base URL>/v1/chat/completions`). The endpoint must live on your verified email's domain — the same anti-squatting rule as claiming — and registrations go through the same review. Once approved, your carrier appears on the public providers and models pages and its listings route like any other provider's.

## Listing your fleet [#listing-your-fleet]

Once your claim is active, register models under **Fleet**: model name, display name, description, context size, max output, capability flags (streaming, vision, audio input, tools, JSON output, reasoning), and the supported `reasoning_effort` tiers. A new listing is created as a draft together with an **initial price filing**; when the LLM Gateway team approves that filing, the model goes active. Everything except pricing stays freely editable afterward.

Carriers claiming a catalogue provider can also **import their catalogue models** in one click: each active catalogue model becomes a managed listing with its current price as an approved filing. Routing keeps using the catalogue entry until the LLM Gateway team retires it — from then on the Airside listing (and its filed fares) takes over. This is how a provider's models migrate from the static catalogue into carrier self-management.

## Price filings [#price-filings]

Listed prices are what developers are billed, so pricing never changes silently: it only changes through a reviewed **price filing**.

* The **initial** filing sets the launch pricing and activates the model on approval.
* Every later price change is an **update** filing — drafted by you, approved by the LLM Gateway team, and only then in effect.

A filing covers input, output, optional cached-input, and optional per-request prices (USD per token, decimal or exponent notation), plus a note for the reviewer. The latest approved filing is the model's effective price; pending and rejected filings are tracked under **Filings**.

## Routing and billing [#routing-and-billing]

Approved listings are genuinely routable. A request for `<provider>/<model>` that does not match the static catalogue resolves against active Airside listings with an approved price filing, and is billed at the filed prices — no extra configuration on the developer side.

## Traffic [#traffic]

The **Traffic** page shows usage of your claimed providers: requests, errors, input/output/total tokens, and billed traffic in USD, as a daily series and broken down per model.

## Fares and routing [#fares-and-routing]

Under **Fares** each claimed provider has two knobs:

* **Traffic discount** (0–50%) — a discount you offer on routed traffic.
* **Landing fee** (5–50%) — the gateway margin you accept; the baseline is &#x2A;*20%**.

Accepting more than the baseline margin or offering a discount makes your traffic effectively cheaper, which boosts you in the [smart routing election](https://docs.llmgateway.io/features/routing); accepting less prices you up. Like price changes, fare changes are filed for review and only reach routing once the LLM Gateway team approves them. The election scores every candidate provider on weighted factors — price after your discount and margin (`0.6`), availability/uptime (`0.5`), prompt-cache support (`0.2`), throughput (`0.05`), and latency (`0.025`) — and the lowest score wins.

<Callout type="info">
  Routing reacts to live metrics: reliability and speed still matter. A cheap
  but flaky deployment loses the election to a slightly pricier, stable one.
</Callout>

## Carrier Slack channel [#carrier-slack-channel]

Every new carrier is invited to a shared cross-team Slack Connect channel with the LLM Gateway team — the fastest way to reach us about claims, filings, or routing.


# Anthropic API Compatibility
URL: https://docs.llmgateway.io/features/anthropic-endpoint


LLMGateway provides a native Anthropic-compatible endpoint at `/v1/messages` that allows you to use any model in our catalog while maintaining the familiar Anthropic API format
This is especially useful for applications designed for Claude that you want to extend to use other models.

<Callout type="info">
  Enjoy a 50% discount on our Anthropic models for a limited time.
</Callout>

## Overview [#overview]

The Anthropic endpoint transforms requests from Anthropic's message format to the OpenAI-compatible format used by LLMGateway, then transforms the responses back to Anthropic's format. This means you can:

* Use **any model** available in LLMGateway with Anthropic's API format
* Maintain existing code that uses Anthropic's SDK or API format
* Access models from OpenAI, Google, Cohere, and other providers through the Anthropic interface
* Leverage LLMGateway's routing, caching, and cost optimization features

## Basic Usage [#basic-usage]

## Configuration for Claude Code [#configuration-for-claude-code]

This endpoint is perfect for configuring Claude Code to use any model available in LLMGateway:

```bash
export ANTHROPIC_BASE_URL=https://api.llmgateway.io
export ANTHROPIC_AUTH_TOKEN=llmgtwy_your_api_key_here
# optional: specify a model, otherwise it uses the default Claude model
export ANTHROPIC_MODEL=gpt-5  # or any model from our catalog

# now run claude!
claude
```

<Callout type="info">
  Environment variables are read once at startup. The `/model` picker lists
  Claude models only, so non-Claude models are selected with `ANTHROPIC_MODEL`
  or `--model`. See the [Claude Code guide](https://docs.llmgateway.io/guides/claude-code) for the
  settings-file options and gateway model discovery.
</Callout>

### Choosing Models [#choosing-models]

You can use any model from the [models page](https://llmgateway.io/models). Popular options for Claude Code include:

```bash
# Use OpenAI's latest model
export ANTHROPIC_MODEL=gpt-5

# Use a cost-effective alternative
export ANTHROPIC_MODEL=gpt-5-mini

# Use Google's Gemini
export ANTHROPIC_MODEL=gemini-3.1-pro-preview

# Use Anthropic's actual Claude models
export ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
```

## Environment Variables [#environment-variables]

When configuring Claude Code or other Anthropic-compatible applications, you can use these environment variables:

### ANTHROPIC\_MODEL [#anthropic_model]

Specifies the main model to use for primary requests.

* **Default**: `claude-sonnet-4-20250514`
* **Example**: `export ANTHROPIC_MODEL=gpt-5`

### ANTHROPIC\_SMALL\_FAST\_MODEL [#anthropic_small_fast_model]

Specifies a smaller, faster model used for background functionality and internal operations.

* **Default**: `claude-3-5-haiku-20241022`
* **Example**: `export ANTHROPIC_SMALL_FAST_MODEL=gpt-5-nano`

```bash
# Example configuration
export ANTHROPIC_BASE_URL=https://api.llmgateway.io
export ANTHROPIC_AUTH_TOKEN=llmgtwy_your_api_key_here
export ANTHROPIC_MODEL=gpt-5
export ANTHROPIC_SMALL_FAST_MODEL=gpt-5-nano
```

## Advanced Features [#advanced-features]

### Making a manual request [#making-a-manual-request]

```bash
curl -X POST "https://api.llmgateway.io/v1/messages" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "max_tokens": 100
  }'
```

### Response Format [#response-format]

The endpoint returns responses in Anthropic's message format:

```json
{
	"id": "msg_abc123",
	"type": "message",
	"role": "assistant",
	"model": "gpt-5",
	"content": [
		{
			"type": "text",
			"text": "Hello! I'm doing well, thank you for asking. How can I help you today?"
		}
	],
	"stop_reason": "end_turn",
	"stop_sequence": null,
	"usage": {
		"input_tokens": 13,
		"output_tokens": 20
	}
}
```

### Request Format [#request-format]

`/v1/messages` expects Anthropic Messages requests and always answers in Anthropic's format. Because the two formats share `model` and `messages`, an OpenAI Chat Completions body can reach this endpoint by accident — and since unknown parameters are ignored rather than rejected, the request succeeds and returns an Anthropic response body that OpenAI SDKs cannot read. If your client reports an empty completion here, check that it is pointed at `/v1/chat/completions`.

Unknown parameters are deliberately ignored rather than rejected, so a valid Anthropic request is never denied for carrying an extra field. As a consequence, OpenAI-only parameters (`response_format`, `stream_options`, `max_completion_tokens`, `n`, `stop`, `seed`, `frequency_penalty`, and similar) have no effect here — the model will not honour them. Use `/v1/chat/completions` if you need them.

A body that is *structurally* OpenAI is rejected by the schema, as it always has been — OpenAI-shaped `tools` (`{"type": "function", "function": {…}}`), OpenAI content parts such as `image_url`, or assistant turns with `content: null`. Those rejections now name the mismatch and point at the right endpoint instead of returning an opaque validation error:

```json
{
	"type": "error",
	"error": {
		"type": "invalid_request_error",
		"message": "This endpoint implements Anthropic's Messages API, and the request body uses OpenAI Chat Completions structures (tools[0].function) that Anthropic's format has no equivalent for. Send OpenAI-format requests to /v1/chat/completions instead, or convert the body to Anthropic's Messages format."
	}
}
```

Rejected requests are recorded in your logs with a `client_error` finish reason and zero cost, so a malformed client is visible in the activity feed rather than failing silently.

### Prompt Caching [#prompt-caching]

For Claude models, `cache_control` markers on `system` and message content blocks are forwarded to the provider unchanged, including the optional `ttl` (`5m` or `1h`):

```json
{
	"model": "claude-sonnet-4-6",
	"max_tokens": 100,
	"system": [
		{
			"type": "text",
			"text": "<several thousand tokens of stable instructions...>",
			"cache_control": { "type": "ephemeral" }
		}
	],
	"messages": [{ "role": "user", "content": "Hello!" }]
}
```

Cache usage comes back in Anthropic's native fields: `usage.cache_creation_input_tokens` (tokens written to the cache this request, billed at the write premium), `usage.cache_read_input_tokens` (tokens served from cache at the discounted rate), and `usage.cache_creation` (the per-TTL write breakdown).

<Callout type="warning">
  Each Claude model has a minimum cacheable prompt length (4,096 tokens on
  current-generation models such as Opus 4.5+, Sonnet 5, and Haiku 4.5). A
  `cache_control` marker on a shorter prompt is accepted but silently not cached
  — both cache usage fields stay `0`. See [Provider Cache
  Control](https://docs.llmgateway.io/features/caching/provider-cache-control) for the per-model
  thresholds and details.
</Callout>

### Web Search [#web-search]

Anthropic's server-side web search tool works on this endpoint. Pass it as usual and the response carries `server_tool_use` and `web_search_tool_result` blocks before the text that cites them, so Anthropic SDK clients surface sources:

```json
{
	"model": "claude-haiku-4-5",
	"max_tokens": 400,
	"messages": [{ "role": "user", "content": "What shipped in Node 24?" }],
	"tools": [{ "type": "web_search_20250305", "name": "web_search" }]
}
```

Replaying the assistant turn verbatim on the next request is supported: the `server_tool_use` and `web_search_tool_result` blocks are accepted and dropped, since the provider re-runs the search rather than reusing the previous results.

### Tool Search [#tool-search]

Anthropic's server-side [tool search](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) works on this endpoint. Pass a `tool_search_tool_*` tool alongside your catalog and mark the tools that should load on demand with `defer_loading: true`:

```json
{
	"model": "claude-sonnet-4-6",
	"max_tokens": 1024,
	"messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
	"tools": [
		{
			"type": "tool_search_tool_regex_20251119",
			"name": "tool_search_tool_regex"
		},
		{
			"name": "get_weather",
			"description": "Get the weather at a specific location",
			"input_schema": { "type": "object" },
			"defer_loading": true
		}
	]
}
```

Deferred tools stay out of the rendered tools section, so adding one does not invalidate an existing prompt cache. The response carries the `server_tool_use` and `tool_search_tool_result` blocks; replay them verbatim on the next request and Anthropic keeps expanding the `tool_reference` entries they carry, so Claude reuses a discovered tool instead of searching again. `tool_reference` blocks returned from your own client-side search inside a `tool_result` are forwarded unchanged too.

Send every tool definition on every request, including the deferred ones — Anthropic needs them server-side to run the search. At least one tool must stay non-deferred (normally the tool search tool itself), and a tool cannot carry both `defer_loading: true` and `cache_control`.

**Where it works.** Tool search reaches the provider on the Anthropic API and on Anthropic models served through Google Cloud, and it needs a Claude 4.5-generation model or newer — older Claude models reject it upstream. On every other provider, including Anthropic models on AWS Bedrock, the tool search tool and `defer_loading` are dropped and all tools are sent eagerly. The request still succeeds, it just loses the cache and token savings, so pin the provider (`anthropic/claude-sonnet-4-6`) when those savings matter.

<Callout type="info">
  Bedrock is a transport limitation rather than a missing capability: Anthropic
  exposes server-side tool search there only through the InvokeModel API, and
  the gateway routes Bedrock through the Converse API.
</Callout>

### Gateway Response Cache [#gateway-response-cache]

If [gateway caching](https://docs.llmgateway.io/features/caching/gateway-caching) is enabled on the project, a byte-identical request is replayed from cache instead of being sent upstream. Because the replayed body is identical (same `id`, same `usage`), the `x-llmgateway-cache: HIT` response header is the marker to check. Send `x-no-cache: true` to bypass the cache for a single request.


# API Keys & IAM Rules
URL: https://docs.llmgateway.io/features/api-keys


API keys are the primary method for authenticating with the LLM Gateway. This guide covers creating API keys, managing them, and configuring IAM rules for fine-grained access control.

## Overview [#overview]

LLM Gateway provides comprehensive API key management with the following features:

* **Basic API Key Management**: Create, list, rename, update, and delete API keys
* **Usage Limits**: Set lifetime and recurring spending limits on individual API keys
* **Expiration (TTL)**: Give a key a time-to-live so it disables itself automatically
* **Rotation (Rolling)**: Replace a key's secret in place without losing its settings or history
* **IAM Rules**: Fine-grained access control for models, providers, and pricing
* **Usage Tracking**: Monitor API key usage and costs
* **Status Management**: Enable/disable keys without deletion

### Related key types [#related-key-types]

This page covers gateway API keys (`llmgtwy_…`), the keys you send to the
gateway as a bearer token. LLM Gateway also issues three other kinds of keys,
each with its own page:

| Key                                                   | What it is for                                                                         |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------- |
| [Master keys](https://docs.llmgateway.io/features/master-keys)                  | Manage projects, gateway API keys, IAM rules, and custom models programmatically       |
| [Provider keys](https://docs.llmgateway.io/learn/provider-keys)                 | Bring your own upstream provider credentials so requests bill to your provider account |
| [Platform secret keys](https://docs.llmgateway.io/features/embeddable-payments) | Mint end-user sessions from your backend when using the Payments SDK (`sk_…`)          |

## Creating API Keys [#creating-api-keys]

### Via Dashboard [#via-dashboard]

1. Navigate to your project in the LLM Gateway dashboard
2. Go to the **API Keys** section
3. Click **Create API Key**
4. Provide a description for your key
5. Optionally set an all-time usage limit
6. Optionally set a recurring usage limit such as `$10 / day` or `$500 / month`
7. Optionally set an expiration (TTL) such as `30 minutes`, `12 hours`, or `7 days`
8. Click **Create**

<Callout type="warning">
  API keys are shown in full only once during creation. Make sure to copy and
  store them securely. New and rolled secrets are stored only as keyed
  HMAC-SHA-256 fingerprints, and authentication compares the fingerprint of the
  presented secret.
</Callout>

### Programmatically [#programmatically]

Gateway API keys can also be created, listed, updated, and deleted without the
dashboard using a [master key](https://docs.llmgateway.io/features/master-keys) — useful when you provision
a key per customer or per environment from your own backend:

```bash
curl -X POST https://internal.llmgateway.io/v1/master/keys \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "proj_...",
    "description": "Customer ACME — production key",
    "periodUsageLimit": "10.00",
    "periodUsageDurationValue": 1,
    "periodUsageDurationUnit": "day"
  }'
```

Usage limits and IAM rules can be configured through the master key API as well.
Expiration (TTL) and rotation are currently dashboard-only. See the [master key
API reference](https://docs.llmgateway.io/features/master-keys#create-a-gateway-api-key) for the full
endpoint list.

## Using API Keys [#using-api-keys]

Once you have an API key, use it in the `Authorization` header of your requests:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer llmgtwy_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

To show DevPass allowance meters in an integration, see the [DevPass Usage
API](https://docs.llmgateway.io/developers/devpass-usage).

## Renaming API Keys [#renaming-api-keys]

A key's name is only a label, so you can change it at any time — from the
dashboard, or with the `description` field on [`PATCH
/v1/master/keys/{id}`](https://docs.llmgateway.io/features/master-keys#update-a-gateway-api-key) — without
affecting the secret, its usage history, limits, or IAM rules.

## Disabling/Enabling API Keys [#disablingenabling-api-keys]

You can disable an API key to stop it from being used, but the key is not deleted and can be re-enabled later.

## Rotating (Rolling) API Keys [#rotating-rolling-api-keys]

Rolling a key generates a **new secret for the same key** and invalidates the old
one immediately. Everything else about the key is preserved: its name, usage
history and statistics, all-time and recurring limits (including the active
period window), IAM rules, and expiration.

Use this when a secret may have been exposed — in a commit, a log, a CI
artifact, or a shared environment — and you want to cut off the leaked value
without losing the key's spend tracking or access rules.

1. Open the **API Keys** page and pick the key's actions menu
2. Choose **Roll Key** and confirm
3. Copy the new secret and update every client that used the old one

<Callout type="warning">
  The old secret stops working the moment the key is rolled, and the new secret
  is shown only once. Roll during a window where you can update your clients
  promptly.
</Callout>

Requests made with the old secret are rejected with a `401 Unauthorized`.
Rolling is limited to regular gateway API keys — the auto-generated playground
key cannot be rolled, and Payments SDK [platform secret
keys](https://docs.llmgateway.io/features/embeddable-payments) have their own lifecycle.

## Expiration (TTL) [#expiration-ttl]

You can give an API key a &#x2A;*time-to-live (TTL)** when you create it. Set how long
the key should live — in **minutes**, **hours**, or **days** — and it will be
disabled automatically once that time passes. This is ideal for short-lived
integrations, demos, CI jobs, and temporary access.

* A key works normally until its expiration time
* Once expired, the gateway rejects requests with that key with a `401 Unauthorized`
* A background job marks expired keys as **inactive**, so the dashboard reflects
  the disabled state
* Keys created without a TTL never expire (the default)

### Reactivating an Expired Key [#reactivating-an-expired-key]

An expired key is paused, not deleted. To bring it back online you must reactivate
it **with a new future expiration** — an expired key cannot be re-enabled while its
TTL is still in the past. Keys that have no TTL, or whose TTL is still in the
future, can be enabled and disabled freely without setting a new expiration.

<Callout type="info">
  Expiration is independent of usage limits. A key can hit its TTL before, or
  instead of, reaching a spend cap.
</Callout>

## Usage Limits [#usage-limits]

Usage is tracked per API key on the API Keys page. Usage includes both costs
from LLM Gateway credits and usage from your own provider keys when applicable,
giving you complete visibility into total spending per key.

You can set two independent limits for each key:

* **All-time usage limit**: A lifetime spend cap
* **Recurring usage limit**: A spend cap that resets every configured hour, day, week, or month

When a key reaches either limit, requests using that key return `401
Unauthorized` until the key is updated or, for recurring limits, the next
usage window starts. This is separate from IAM rule violations, which return
`403 Forbidden`.

Recurring windows support:

* Minimum duration: **1 hour**
* Maximum duration: **12 months**
* Units: **hour**, **day**, **week**, **month**

For the dashboard walkthrough and field-by-field details, see [API Keys in
Learn](https://docs.llmgateway.io/learn/api-keys).

## IAM Rules [#iam-rules]

IAM (Identity Access Management) rules provide fine-grained access control over what models, providers, and pricing tiers an API key can access.

### Rule Types [#rule-types]

#### Model Access Rules [#model-access-rules]

Control access to specific models:

* **Allow Models**: Only allow access to specific models
* **Deny Models**: Block access to specific models

#### Provider Access Rules [#provider-access-rules]

Control access to specific providers:

* **Allow Providers**: Only allow access to specific providers
* **Deny Providers**: Block access to specific providers

Provider rules can also target your organization's own [custom providers](https://docs.llmgateway.io/features/custom-providers). The generic `custom` entry matches every custom provider, while a `custom:<name>` entry (offered in the selector for each of your custom providers) matches only the custom provider with that name. Model rules can likewise reference a custom-catalog model as `<name>/<model>`.

#### Pricing Rules [#pricing-rules]

Control access based on model pricing:

* **Allow Pricing**: Set constraints on what pricing tiers are allowed
* **Deny Pricing**: Block specific pricing tiers
* **Free vs Paid**: Allow or deny access to free vs paid models

#### IP Address Rules [#ip-address-rules]

<Callout type="info">
  IP address rules are available on the **Enterprise** plan only. Contact us at
  [contact@llmgateway.io](mailto:contact@llmgateway.io) to enable them for your organization.
</Callout>

Restrict where the API key can be used from by source IP, using CIDR ranges:

* **Allow IP Ranges (CIDR)**: Only permit requests from the listed IPv4/IPv6 CIDRs
* **Deny IP Ranges (CIDR)**: Block requests from the listed IPv4/IPv6 CIDRs

Both IPv4 (e.g. `192.0.2.0/24`) and IPv6 (e.g. `2001:db8::/32`) ranges are supported, and you can mix both in a single rule. To restrict to a single address, use a `/32` (IPv4) or `/128` (IPv6) prefix.

The gateway reads the client IP from the first entry in the `X-Forwarded-For` header (set by the GCP load balancer). When an `allow_ip_cidrs` rule is configured and the gateway cannot determine the client IP, the request is denied. Invalid CIDR syntax is rejected at rule-creation time with a `400` error.

### Combining Multiple Rules [#combining-multiple-rules]

* **Allow rules of the same type are unioned**: a request passes if it matches *any* of them. For example, one `allow_models` rule with `["claude-opus-4-6"]` and another with `["claude-fable-5"]` allow both models — exactly as if you had a single rule listing both.
* **Allow rules of different types are combined with AND**: the request must satisfy every configured allow rule type (e.g. the model must be in the allowed models *and* served by an allowed provider).
* **Deny rules always apply**: a request matching any deny rule is rejected, regardless of allow rules.
* **Moderation is special-cased**: `/v1/moderations` runs a fixed moderation model that cannot appear in model allowlists, so only provider and IP rules apply to it — model and pricing rules are skipped.

## Member-Level IAM Rules [#member-level-iam-rules]

The same rule types can also be configured **per organization member** by owners and admins on the [Team page](https://docs.llmgateway.io/learn/team) (admins cannot modify an owner's rules). Member-level rules act as an organization-wide ceiling for that member:

* A request must pass **both** the member's rules and the API key's rules. Within each level, rules combine exactly as described above.
* Key rules can only **narrow** access further — they can never grant anything the member's rules deny. For example, if an admin restricts a member to a single approved [provider](https://llmgateway.io/providers) with an `allow_providers` rule, the member can create a key rule allowing only a specific model from that provider, but a key rule allowing any other provider has no effect.
* A key with **no rules of its own** is still fully constrained by its owner's member-level rules.
* Member-level rules apply to all regular API keys created by that member, across every project in the organization.

When a request is denied by a member-level rule, the `403` error message states that the restriction is an organization member IAM rule set by the org admin (rather than the key's own IAM configuration), so key holders know who to contact.

Member-level rules can also be managed programmatically via the [master key API](https://docs.llmgateway.io/features/master-keys#member-iam-rules), addressing members by membership id or email.

## Error Handling [#error-handling]

When API keys encounter IAM rule violations, the API returns a `403` with the standard OpenAI error envelope:

```json
{
	"error": {
		"message": "Access denied: Model gpt-4 is not in the allowed models list",
		"type": "invalid_request_error",
		"param": null,
		"code": "permission_denied"
	}
}
```

Common error scenarios:

* Model not allowed by IAM rules
* Provider blocked by IAM rules
* Pricing limits exceeded
* API key disabled or deleted
* API key expired (TTL passed)
* API key rolled, so the old secret is no longer valid
* Usage limit reached

## Migration from Legacy Keys [#migration-from-legacy-keys]

If you have existing API keys without IAM rules:

1. **Backward Compatibility**: Existing keys continue to work without restrictions
2. **Gradual Migration**: Add IAM rules incrementally
3. **Testing**: Test IAM rules in development before applying to production
4. **Monitoring**: Monitor for access denied errors after implementing rules

<Callout type="info">
  API keys without IAM rules have unrestricted access to all models and
  providers.
</Callout>


# Audit Logs
URL: https://docs.llmgateway.io/features/audit-logs


Audit logs provide complete visibility into all actions within your organization. Track who did what, when, and to which resource.

<Callout type="info">
  Audit logs are available on the [**Enterprise
  plan**](https://llmgateway.io/enterprise) for organization owners and admins.
</Callout>

## What's Tracked [#whats-tracked]

Every significant action is logged with detailed metadata:

| Field             | Description                                              |
| ----------------- | -------------------------------------------------------- |
| **Timestamp**     | When the action occurred                                 |
| **User**          | Who performed the action (name and email)                |
| **Action**        | What was done (e.g., `api_key.create`, `project.update`) |
| **Resource Type** | Category of the affected resource                        |
| **Resource ID**   | Unique identifier of the affected resource               |
| **Details**       | Additional context like resource names or changed fields |

## Tracked Actions [#tracked-actions]

### Organization Management [#organization-management]

* `organization.update` — Organization settings changed
* `organization.delete` — Organization deleted

### Project Management [#project-management]

* `project.create` — New project created
* `project.update` — Project settings changed
* `project.delete` — Project deleted

### Team Management [#team-management]

* `team_member.add` — New member invited
* `team_member.update` — Member role changed
* `team_member.remove` — Member removed

### API Key Management [#api-key-management]

* `api_key.create` — New API key created
* `api_key.update_status` — API key enabled/disabled
* `api_key.update_limit` — Usage limit changed
* `api_key.delete` — API key deleted
* `api_key.iam_rule.create` — IAM rule added
* `api_key.iam_rule.update` — IAM rule modified
* `api_key.iam_rule.delete` — IAM rule removed

### Provider Key Management [#provider-key-management]

* `provider_key.create` — Provider key added
* `provider_key.update` — Provider key status changed
* `provider_key.delete` — Provider key removed

### Billing Events [#billing-events]

* `subscription.create` — Subscription started
* `subscription.cancel` — Subscription cancelled
* `subscription.resume` — Subscription resumed
* `payment.credit_topup` — Credits purchased

## Filtering and Search [#filtering-and-search]

Filter logs by:

* **Action** — Specific action type
* **Resource Type** — Category of resource
* **User** — Who performed the action
* **Date Range** — Time period

## Data Retention [#data-retention]

Audit logs are retained for **90 days** on the Enterprise plan.

## Access Control [#access-control]

Only organization **owners** and **admins** can view audit logs. This ensures sensitive activity data is only visible to authorized personnel.

## Get Started [#get-started]

Audit logs are an Enterprise feature. [Contact us](https://llmgateway.io/enterprise) to enable Enterprise for your organization.


# Compliance
URL: https://docs.llmgateway.io/features/compliance


Provider compliance policies let you guarantee that requests are only ever routed to providers that meet your organization's regulatory requirements. When a request would be routed to a provider that doesn't meet the policy, the gateway blocks it **before any data leaves the gateway**.

<Callout type="info">
  The complete provider compliance policy is available on
  [**Enterprise**](https://llmgateway.io/enterprise). DevPass offers only **No
  AI training**.
</Callout>

## Requirements [#requirements]

Enable a policy under **Settings → Compliance** in the dashboard and toggle the requirements you need:

In DevPass, enable **No AI training** directly under **Settings**. It uses the same fail-closed provider policy described below, but the other requirements are not available on DevPass.

| Requirement                   | A provider is allowed when…                                      |
| ----------------------------- | ---------------------------------------------------------------- |
| **Zero data retention (ZDR)** | it does not log prompts and declares a zero-day retention period |
| **No training on prompts**    | it does not train on API prompts                                 |
| **No stealth providers**      | it is not a stealth provider                                     |
| **GDPR compliant**            | it is GDPR compliant                                             |
| **SOC 2 (Type 1 or 2)**       | it holds a SOC 2 report of any type                              |
| **SOC 2 Type 2**              | it holds a SOC 2 Type 2 report specifically                      |
| **ISO 27001**                 | it holds an ISO 27001 certification                              |
| **SOC 2 Type 2 or ISO 27001** | it holds either a SOC 2 Type 2 report or ISO 27001               |

Every requirement is **fail-closed**: a provider passes only if its published data policy explicitly satisfies the requirement. If an attribute is unknown for a provider, that provider is treated as non-compliant.

Before enabling &#x2A;*Zero data retention (ZDR)**, set the organization's data retention to **Metadata Only** and disable response caching in every project. While ZDR is active, **Retain All Data** and project response caching are unavailable, gateway response caching is bypassed, provider prompt-cache markers are stripped, Responses API requests must set `store: false`, and asynchronous video generation is unavailable because video jobs require temporary output storage. Disable ZDR before turning payload retention or caching back on.

Existing policies that use the deprecated **No prompt logging** rule continue to enforce their original provider-routing behavior. The legacy rule is no longer available for new configuration and does not enable the gateway-level ZDR controls described above.

Stealth providers are undisclosed platforms that LLM Gateway routes to without naming the operator, so their data policy and headquarters are unknown. That already makes them fail every certification and data-policy requirement above, but **No stealth providers** excludes them explicitly — useful when you want them gone without turning on any other requirement.

The settings page shows a live **Provider Impact** preview of which providers are allowed (green) and which are blocked (red) under the current policy, so you can see the impact before saving. Hovering a blocked provider lists exactly which requirements it does not meet — a missing certification, its data policy, its headquarters country, or a provider-list restriction.

## Provider headquarters [#provider-headquarters]

Beyond certifications, you can restrict routing by the country a provider is headquartered in. The **Provider Headquarters** card presents a country selector, and requests are only routed to providers based in a selected country.

The selector only offers countries that are referenced by a provider in the catalogue — browse the [providers directory](https://llmgateway.io/providers) to see the current set and each provider's headquarters. Selecting no country applies no location restriction. The country filter is also fail-closed: when at least one country is selected, a provider whose headquarters is unknown is treated as non-compliant.

The country filter composes with the certification and data-policy requirements above — a provider must satisfy all active requirements to be allowed.

## Provider & model restrictions [#provider--model-restrictions]

Beyond attribute-based requirements, the **Provider & Model Restrictions** card lets you block or allow individual providers and models:

* **Blocked providers / blocked models** — deny lists. A listed provider or model is always blocked, even when it satisfies every requirement above.
* **Allowed providers / allowed models** — fine-grained allow lists. When non-empty, only the listed providers (or models) may be used; everything else is blocked. An empty list applies no restriction.

Deny lists always win over allow lists, and both compose with the requirements above — an allow-listed provider must still satisfy every active certification, data-policy, and country requirement.

<Callout type="warn">
  A non-empty allowed-providers list blocks **every** provider not on it,
  regardless of certifications or data policy. If you allow-list only a custom
  provider, all catalogue providers show as blocked with the reason "Not on the
  allowed-providers list" — add any additional provider you want to use to the
  allow list.
</Callout>

The selectors include your organization's own [custom providers](https://docs.llmgateway.io/features/custom-providers) (stored as `custom:<name>` refs) and their custom-catalog models (stored as `<name>/<model>` refs), so a specific custom deployment can be blocked or allow-listed individually just like a catalogue provider.

### Choosing a compliant provider [#choosing-a-compliant-provider]

The provider dropdowns are policy-aware, so you can tell **before** adding a provider to a list whether it satisfies your policy:

* A **green shield** marks a provider that meets every active certification, data-policy, and headquarters requirement.
* A **red shield** marks a provider that does not; the requirements it misses (for example "May log prompts" or "Headquartered in China, which is not an allowed country") are listed under its name.
* The &#x2A;*"Only providers that meet policy requirements"** toggle at the top of the dropdown hides incompatible providers entirely.

These indicators evaluate the requirements only — deliberately ignoring the allowed/blocked lists themselves — so an active allow list never paints every other provider red while you're deciding what to add to it. Your own custom providers are evaluated against their [self-attested posture](#custom-providers); one without an attestation on file shows "No compliance attestation on file".

Elsewhere in the dashboard (for example the API-key provider filter), the colored dot next to a provider is simply the provider's **brand color** and carries no compliance meaning.

These restrictions are **organization-wide** and take precedence over member-level and API-key-level [IAM rules](https://docs.llmgateway.io/features/api-keys): they are enforced after IAM evaluation, so no user-, team-, or key-level allow rule can grant access to a provider or model the compliance policy excludes.

## Enforcement [#enforcement]

When no available provider for a model meets the policy — or a pinned provider is non-compliant — the gateway returns a `403`:

```json
{
	"error": {
		"message": "This request was blocked by your organization's provider compliance policy. No available provider for deepseek-v3.2 meets the required certifications or provider/model restrictions. Contact your LLMGateway admin to adjust the policy."
	}
}
```

This applies to both routing modes:

* **Automatic routing** — non-compliant providers are removed from the candidate set, and the request is blocked if none remain.
* **Pinned providers** — a request such as `deepseek/deepseek-v3.2` is blocked when that specific provider does not meet the policy.

Each block is recorded as a **security event** so administrators can review what was rejected and why.

## Custom providers [#custom-providers]

[Custom providers](https://docs.llmgateway.io/features/custom-providers) point at infrastructure your organization operates itself (for example, models hosted in your own cloud account), so they have no entry in the provider catalogue and no published data policy. Under any enabled compliance policy they are therefore **blocked by default** — the same fail-closed rule that applies to catalogue providers with unknown attributes.

To route through a custom provider while a policy is active, an organization owner or admin can record a **self-attestation** of that provider key's compliance posture under **Models → Compliance attestation** in the dashboard. The attestation covers the same attributes as a catalogue data policy — SOC 2 report type, ISO 27001, GDPR, training on API prompts, prompt logging, retention period, and the country the deployment is operated from — and the gateway evaluates it against your policy with **identical fail-closed rules**: an attribute left "unknown" never satisfies a requirement, and only an explicit "No" satisfies requirements like "No training on prompts". Clearing the attestation restores the blocked state.

<Callout type="warn">
  LLMGateway does not verify attestations — they are your organization's own
  assertion about infrastructure you operate. Every attestation change is
  recorded in the audit log with the attesting user and timestamp.
</Callout>

Attestations only apply to custom provider keys. They can never be used to clear a catalogue provider that fails your policy.

**Known limitation:** the country selector in the compliance policy only offers countries referenced by catalogue providers (see the [providers directory](https://llmgateway.io/providers)). A custom deployment attested to a country outside that set cannot be allowed while a country restriction is active.

## Safety identifiers [#safety-identifiers]

Providers that offer an abuse-attribution identifier receive an opaque, random value that LLM Gateway generates once per organization. It contains no personal data — no email address, user id or organization name — and it is the only thing that ties a report of abusive traffic back to an account.

Whether LLM Gateway forwards this identifier depends on the upstream API. Check the [providers directory](https://llmgateway.io/providers) and open a provider to see its current behavior. Any `safety_identifier` you set on a request yourself is ignored — the value is always the one derived from your organization.

## Access Control [#access-control]

Only organization **owners** and **admins** can view and change the compliance policy.

Project-scoped **developer** members cannot see the policy itself, but they can browse the org's available providers and models — including custom providers and their model catalogs — read-only on the **Models** page. This is the recommended way for developers to discover what they can call without being granted additional permissions.

## Related [#related]

* [Org Models Directory](https://docs.llmgateway.io/features/models-directory) — see per-model eligibility under the active policy in the dashboard.
* [Data Retention](https://docs.llmgateway.io/features/data-retention) — control whether request and response payloads are stored.
* [Guardrails](https://docs.llmgateway.io/features/guardrails) — detect and block harmful or sensitive content.

## Get Started [#get-started]

DevPass subscribers can configure **No AI training** under **Settings**. [Contact us](https://llmgateway.io/enterprise) for the complete provider compliance policy.


# Cost Breakdown
URL: https://docs.llmgateway.io/features/cost-breakdown


LLM Gateway provides real-time cost information for each API request directly in the response's `usage` object. This allows you to track costs programmatically without needing to query the dashboard.

<Callout type="info">
  Cost breakdown is available for all users on both hosted and self-hosted
  deployments.
</Callout>

## Response Format [#response-format]

API responses include cost fields in the `usage` object:

```json
{
	"id": "chatcmpl-123",
	"object": "chat.completion",
	"created": 1234567890,
	"model": "openai/gpt-4o",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "Hello! How can I help you today?"
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 10,
		"completion_tokens": 15,
		"total_tokens": 25,
		"cost": 0.000125,
		"cost_details": {
			"upstream_inference_cost": 0.000125,
			"upstream_inference_prompt_cost": 0.000025,
			"upstream_inference_completions_cost": 0.0001,
			"total_cost": 0.000125,
			"input_cost": 0.000025,
			"output_cost": 0.0001,
			"cached_input_cost": 0,
			"request_cost": 0,
			"web_search_cost": 0,
			"image_input_cost": null,
			"image_output_cost": null,
			"data_storage_cost": 0.00000025
		},
		"prompt_tokens_details": {
			"cached_tokens": 0,
			"cache_write_tokens": 0,
			"audio_tokens": 0,
			"video_tokens": 0
		},
		"completion_tokens_details": {
			"reasoning_tokens": 0,
			"image_tokens": 0,
			"audio_tokens": 0
		}
	}
}
```

## Cost Fields [#cost-fields]

| Field                                              | Description                                                                                                                                            |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `cost`                                             | Total inference cost for the request in USD                                                                                                            |
| `cost_details.upstream_inference_cost`             | Combined upstream inference cost in USD (prompt + completions)                                                                                         |
| `cost_details.upstream_inference_prompt_cost`      | Upstream cost for prompt tokens in USD (includes cached prompt discount)                                                                               |
| `cost_details.upstream_inference_completions_cost` | Upstream cost for completion tokens in USD                                                                                                             |
| `cost_details.total_cost`                          | Total request cost in USD (LLM Gateway extended field)                                                                                                 |
| `cost_details.input_cost`                          | Cost for non-cached prompt tokens in USD                                                                                                               |
| `cost_details.output_cost`                         | Cost for completion tokens in USD                                                                                                                      |
| `cost_details.cached_input_cost`                   | Cost for cached prompt tokens in USD                                                                                                                   |
| `cost_details.cache_write_input_cost`              | Cost for prompt tokens written to the provider cache in USD, billed at the provider's cache-write premium (e.g. 1.25x for 5m / 2x for 1h on Anthropic) |
| `cost_details.request_cost`                        | Per-request flat fee in USD (when the model applies one)                                                                                               |
| `cost_details.web_search_cost`                     | Cost for web search tool calls in USD                                                                                                                  |
| `cost_details.image_input_cost`                    | Cost for image inputs in USD                                                                                                                           |
| `cost_details.image_output_cost`                   | Cost for image outputs in USD                                                                                                                          |
| `cost_details.data_storage_cost`                   | Storage cost for retained request/response payloads in USD                                                                                             |

## Token Detail Fields [#token-detail-fields]

The `usage` object also includes detailed token counters that mirror OpenAI's extended format:

| Field                                         | Description                                                                                                                        |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `prompt_tokens_details.cached_tokens`         | Number of prompt tokens served from the provider's prompt cache                                                                    |
| `prompt_tokens_details.cache_write_tokens`    | Number of prompt tokens written into the provider's prompt cache                                                                   |
| `prompt_tokens_details.cache_creation_tokens` | Alias of `cache_write_tokens`, matching Anthropic's naming                                                                         |
| `prompt_tokens_details.cache_creation`        | Per-TTL breakdown of cache writes (`ephemeral_5m_input_tokens` / `ephemeral_1h_input_tokens`), present when a cache write occurred |
| `prompt_tokens_details.audio_tokens`          | Number of audio prompt tokens                                                                                                      |
| `prompt_tokens_details.video_tokens`          | Number of video prompt tokens                                                                                                      |
| `completion_tokens_details.reasoning_tokens`  | Number of reasoning tokens produced by reasoning models                                                                            |
| `completion_tokens_details.image_tokens`      | Number of image tokens produced                                                                                                    |
| `completion_tokens_details.audio_tokens`      | Number of audio tokens produced                                                                                                    |

## Streaming Responses [#streaming-responses]

Cost information is also available in streaming responses. The cost fields are included in the final usage chunk sent before the `[DONE]` message:

```
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","choices":[...],"usage":{"prompt_tokens":10,"completion_tokens":15,"total_tokens":25,"cost":0.000125,"cost_details":{"upstream_inference_cost":0.000125,"upstream_inference_prompt_cost":0.000025,"upstream_inference_completions_cost":0.0001,"total_cost":0.000125,"input_cost":0.000025,"output_cost":0.0001,"cached_input_cost":0,"request_cost":0,"web_search_cost":0,"image_input_cost":null,"image_output_cost":null,"data_storage_cost":0.00000025}}}

data: [DONE]
```

## Example: Tracking Costs in Code [#example-tracking-costs-in-code]

Here's an example of how to track costs programmatically using the cost breakdown feature:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
	baseURL: "https://api.llmgateway.io/v1",
});

async function trackCosts() {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages: [{ role: "user", content: "Hello!" }],
	});

	const usage = response.usage as any;

	if (usage.cost !== undefined) {
		console.log(`Request cost: $${usage.cost.toFixed(6)}`);
		console.log(
			`  Prompt: $${usage.cost_details.upstream_inference_prompt_cost.toFixed(6)}`,
		);
		console.log(
			`  Completions: $${usage.cost_details.upstream_inference_completions_cost.toFixed(6)}`,
		);

		const cachedTokens = usage.prompt_tokens_details?.cached_tokens ?? 0;
		if (cachedTokens > 0) {
			console.log(`  Cached prompt tokens: ${cachedTokens}`);
		}
	}

	return response;
}
```

## Use Cases [#use-cases]

### Budget Monitoring [#budget-monitoring]

Track costs in real-time and implement budget limits in your application:

```typescript
let totalSpent = 0;
const BUDGET_LIMIT = 10.0; // $10 budget

async function makeRequest(messages: Message[]) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages,
	});

	const cost = (response.usage as any).cost || 0;
	totalSpent += cost;

	if (totalSpent > BUDGET_LIMIT) {
		throw new Error(`Budget exceeded: $${totalSpent.toFixed(2)}`);
	}

	return response;
}
```

### Per-User Cost Allocation [#per-user-cost-allocation]

Track costs per user for billing or analytics:

```typescript
const userCosts: Map<string, number> = new Map();

async function makeRequestForUser(userId: string, messages: Message[]) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages,
	});

	const cost = (response.usage as any).cost || 0;
	const currentCost = userCosts.get(userId) || 0;
	userCosts.set(userId, currentCost + cost);

	return response;
}
```

### Cost Analytics [#cost-analytics]

Aggregate costs by model, time period, or any other dimension:

```typescript
interface CostEntry {
	timestamp: Date;
	model: string;
	promptCost: number;
	completionsCost: number;
	totalCost: number;
}

const costLog: CostEntry[] = [];

async function loggedRequest(model: string, messages: Message[]) {
	const response = await client.chat.completions.create({
		model,
		messages,
	});

	const usage = response.usage as any;

	costLog.push({
		timestamp: new Date(),
		model: response.model,
		promptCost: usage.cost_details?.upstream_inference_prompt_cost || 0,
		completionsCost:
			usage.cost_details?.upstream_inference_completions_cost || 0,
		totalCost: usage.cost || 0,
	});

	return response;
}
```

## Self-Hosted Deployments [#self-hosted-deployments]

If you're running a self-hosted LLM Gateway deployment, cost breakdown is always included in API responses regardless of plan. This allows you to track internal costs and allocate them across teams or projects.


# Custom Providers
URL: https://docs.llmgateway.io/features/custom-providers


LLMGateway supports integrating custom OpenAI-compatible providers, allowing you to use any API that follows the OpenAI chat completions format. This feature is perfect for:

* Private or self-hosted LLM deployments
* Specialized AI providers not natively supported
* Internal AI services within your organization
* Testing against different model endpoints

<Callout type="info">
  Custom providers must be OpenAI-compatible, supporting the
  `/v1/chat/completions` endpoint format.
</Callout>

## Quick Setup [#quick-setup]

### 1. Add a Custom Provider Key [#1-add-a-custom-provider-key]

Navigate to your organization's provider settings and add a custom provider via the UI.
Provide a lowercase name, OpenAI-compatible base URL, and API token for the custom provider.

### 2. Make Requests [#2-make-requests]

Once configured, make requests using the format `{customName}/{modelName}`:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mycompany/custom-gpt-4",
    "messages": [
      {
        "role": "user",
        "content": "Hello from my custom provider!"
      }
    ]
  }'
```

## Configuration Requirements [#configuration-requirements]

### Custom Provider Name [#custom-provider-name]

* **Format**: Lowercase letters (`a-z`) with optional single hyphens between them
* **Examples**: `mycompany`, `internal`, `my-company`, `eu-west`
* **Invalid**: `MyCompany`, `my_company`, `123test`, `-mycompany`, `my-`, `my--company`

<Callout type="warn">
  The custom provider name must match the regex pattern `/^[a-z]+(-[a-z]+)*$/`
  exactly.
</Callout>

### Base URL [#base-url]

* Must be a valid URL pointing to your provider's base endpoint
* Use an `https://` URL for hosted providers; an `http://` URL is allowed when
  self-hosting (for example a service on your own network)
* LLMGateway will append `/v1/chat/completions` automatically
* **Example**: `https://api.example.com` → `https://api.example.com/v1/chat/completions`

### API Token [#api-token]

* Provider-specific authentication token
* Used in the `Authorization: Bearer {token}` header

<Callout type="info">
  Unlike built-in providers, custom provider models are not validated, giving
  you complete flexibility.
</Callout>

## Supported Features [#supported-features]

Custom providers inherit full LLMGateway functionality.

## Custom Model Catalog [#custom-model-catalog]

<Callout type="info">
  The custom model catalog is an **Enterprise** feature. Managing entries
  requires an enterprise plan, but the gateway always honors catalog entries
  that already exist.
</Callout>

By default, requests through a custom provider are **not billed** and have **no
enforced limits** — LLMGateway has no catalog entry for the model, so it cannot
know its pricing, context window, or capabilities. The **custom model catalog**
lets you define that information per custom provider key, so LLMGateway can
attribute cost, enforce limits, and report usage just like a built-in model.

Custom catalog models are **text-output** models. Multi-modal *input* (images,
audio) is still supported and can be priced via the input fields below; output
generation (images, video, audio) is intentionally out of scope because it is
too provider-specific to bill generically.

### Defining a model [#defining-a-model]

Open **Organization → Models**, pick the custom provider key, and add a
catalog entry. Every field except the model id is optional:

| Field                    | Purpose                                                                                                                         |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| **Model id**             | The id used after the provider prefix (e.g. `gpt-5.5` in `mycompany/gpt-5.5`).                                                  |
| **Display name**         | Optional human-readable label.                                                                                                  |
| **Context size**         | Maximum input window in tokens. Enforced per request.                                                                           |
| **Max output**           | Maximum completion tokens. Enforced against `max_tokens`.                                                                       |
| **Token prices**         | Input, output, cached input, cache read, cache write (5m and 1h), per-request, web search, image input, and audio input prices. |
| **Capabilities**         | `streaming`, `vision`, `tools`, `reasoning`, `jsonOutput`, `audio`.                                                             |
| **Supported parameters** | Advisory list of accepted request parameters.                                                                                   |

Prices are in **USD per token** and accept either decimal (`0.000003`) or
exponent (`3.0e-6`) notation.

### Cost attribution [#cost-attribution]

When a request matches a catalog entry, LLMGateway bills it using the entry's
token prices and records the cost on the activity log. Without a matching entry
the request stays unbilled (zero/null cost).

<Callout type="warn">
  A known model id (for example `gpt-5.5`) routed through a custom provider is
  **not** billed at that model's public pricing — only the prices you define in
  the catalog apply. Define the model in the catalog to attribute cost.
</Callout>

### Limit and capability enforcement [#limit-and-capability-enforcement]

When an entry defines limits or capabilities, LLMGateway enforces them before
forwarding the request:

* **Context size / max output** — requests that would exceed the configured
  window or output budget are rejected with `400`.
* **Capabilities** — when a flag is explicitly set to disabled, requests that use
  that feature are rejected (e.g. images against a non-`vision` model, tools
  against a non-`tools` model, streaming against a non-streaming model). Flags
  left unset stay permissive and the upstream provider enforces them.

### Restricting to the catalog [#restricting-to-the-catalog]

Each custom provider key has an **Only allow catalog models** switch. When
enabled, requests through that provider must reference a defined catalog model —
undefined models are rejected with `400`. This guarantees that every request has
known cost attribution and enforced limits. When disabled, undefined models
still work but remain unbilled, as before.

## Visibility [#visibility]

Every active organization member — including project-scoped **developer**
members — can browse the org's custom providers and their model catalogs
read-only on the **Models** page, so developers can always discover
which providers and models are available to them. Creating, editing, and
deleting providers, models, and attestations stays restricted to organization
owners and admins.

## Compliance attestation [#compliance-attestation]

If your organization enforces a [provider compliance policy](https://docs.llmgateway.io/features/compliance),
custom providers are blocked by default — they have no catalogue data policy, so
the fail-closed rules treat them as non-compliant. Organization owners and
admins can record a per-key **self-attestation** of the deployment's compliance
posture under **Models → Compliance attestation**, which the gateway then
evaluates against the policy exactly like a catalogue provider's data policy.
See [Compliance → Custom providers](https://docs.llmgateway.io/features/compliance#custom-providers) for
details and caveats.


# Data Retention
URL: https://docs.llmgateway.io/features/data-retention


LLM Gateway offers configurable data retention policies that allow you to store full request and response payloads. This enables powerful debugging capabilities, detailed analytics, and compliance with data governance requirements.

## Retention Levels [#retention-levels]

LLM Gateway supports two retention levels that can be configured per organization:

| Level               | Description                                                                                    | Storage Cost    |
| ------------------- | ---------------------------------------------------------------------------------------------- | --------------- |
| **Metadata Only**   | Stores request metadata (timestamps, model, tokens, costs) without full payloads. Default.     | Free            |
| **Retain All Data** | Stores complete request and response payloads including messages, tool calls, and attachments. | $0.01/1M tokens |

<Callout type="info">
  Metadata-only retention is enabled by default and provides usage analytics
  without additional storage costs.
</Callout>

<Callout type="warning">
  Retention levels are configurable on standard (pay-as-you-go) organizations
  only. DevPass and chat subscriptions are always metadata only — their request
  and response payloads are not retained, and there is no setting to turn
  payload storage on. The Responses API exception below still applies to them.
</Callout>

## Storage Pricing [#storage-pricing]

When full data retention is enabled, storage is billed at **$0.01 per 1 million tokens**. This rate applies to:

* Input tokens (prompt)
* Cached input tokens
* Output tokens (completion)
* Reasoning tokens

Storage costs are calculated per request and billed separately from inference. When "Retain All Data" is enabled, each response's `usage.cost_details` object includes a `data_storage_cost` field with the per-request storage cost in USD. See [Cost Breakdown](https://docs.llmgateway.io/features/cost-breakdown) for the full list of cost fields.

### Example Cost Calculation [#example-cost-calculation]

For a request with:

* 1,000 input tokens
* 500 output tokens
* 1,500 total tokens

Storage cost = 1,500 / 1,000,000 × $0.01 = **$0.000015**

## Configuring Retention [#configuring-retention]

Data retention is configured at the organization level in your dashboard settings. The setting is only available on standard pay-as-you-go organizations:

1. Navigate to **Organization Settings** → **Policies**
2. Select your preferred **Data Retention Level**
3. Save changes

<Callout type="warning">
  Changing retention settings applies to new requests only. Existing stored data
  follows the retention period active when it was created.
</Callout>

## Retention Periods [#retention-periods]

Data is retained for 30 days for all users. Enterprise plans can have custom retention periods. After the retention period expires, data is automatically deleted.

<Callout type="info">
  The Responses API does not require data retention. Stored responses (used for
  `previous_response_id` chaining and `GET /v1/responses/:id`) are kept in
  dedicated storage for 30 days — matching OpenAI's own retention — regardless
  of your organization's data retention policy, and are not billed as data
  storage. Because this is independent of the retention level, it applies to
  metadata-only organizations, including DevPass and chat, whose Responses API
  input and output items are therefore held for up to 30 days. Send `store:
  	false` with the request to opt out. When an Enterprise organization's
  [zero-data-retention policy](https://docs.llmgateway.io/features/compliance) is active, the gateway
  rejects Responses API requests unless `store: false` is set, does not retain
  compaction state, bypasses the gateway response cache, strips provider
  prompt-cache markers, and prevents payload retention or project response
  caching from being enabled until ZDR is disabled. ZDR cannot be enabled until
  both settings are off.
</Callout>

## Accessing Stored Data [#accessing-stored-data]

When data retention is enabled, you can access your stored requests through the dashboard:

* View request history with full payload inspection
* Filter by model and date range
* Inspect complete request and response payloads

## Use Cases [#use-cases]

### Debugging [#debugging]

Full data retention enables you to:

* Inspect exact prompts sent to models
* Review complete responses including tool calls
* Trace conversation histories
* Identify issues in production

### Analytics [#analytics]

With stored payloads, you can:

* Analyze prompt patterns and effectiveness
* Track response quality over time
* Build custom dashboards and reports
* Measure model performance across use cases

### Compliance [#compliance]

Data retention helps meet compliance requirements by:

* Maintaining audit trails of AI interactions
* Enabling data governance policies
* Supporting incident investigation
* Providing records for regulatory requirements

## Billing Considerations [#billing-considerations]

### Credit Usage [#credit-usage]

In **API keys mode** (using your own provider keys):

* Only storage costs are deducted from LLM Gateway credits
* Inference costs are billed directly to your provider

In **credits mode**:

* Both inference and storage costs are deducted from credits

### Monitoring Storage Costs [#monitoring-storage-costs]

Storage costs appear in:

* Usage dashboard under "Storage" category
* Billing invoices as a separate line item

<Callout type="success">
  Enable [auto top-up](https://llmgateway.io/dashboard) in billing settings to
  ensure uninterrupted service when storage costs accumulate.
</Callout>

## Self-Hosted Deployments [#self-hosted-deployments]

Self-hosted deployments have full control over data retention:

* Configure retention periods in environment variables
* Data is stored in your own PostgreSQL database
* No additional storage costs (you manage your own infrastructure)

## Privacy and Security [#privacy-and-security]

* All stored data is encrypted at rest
* Access is restricted to organization members with appropriate permissions
* Data is automatically deleted after the retention period
* You can request immediate deletion of specific records through support


# Document Reading
URL: https://docs.llmgateway.io/features/documents


LLMGateway supports sending documents (PDFs and other file types) to document-capable models using OpenAI's `file` content block format. The gateway forwards the document to the underlying provider so the model can read and reason over its contents.

## Document-Capable Models [#document-capable-models]

Document input is currently supported on Google Gemini models via Google AI Studio. You can find document-capable models on the [models page with the document filter](https://llmgateway.io/models?filters=1\&document=true).

## Sending a Document [#sending-a-document]

Add a `file` content block to a user message. The `file_data` field must be a base64-encoded data URL that includes the document's MIME type.

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.6-flash",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Summarize this document."
          },
          {
            "type": "file",
            "file": {
              "filename": "report.pdf",
              "file_data": "data:application/pdf;base64,JVBERi0xLjQKJ..."
            }
          }
        ]
      }
    ]
  }'
```

### Content Block Fields [#content-block-fields]

* **`type`**: must be `"file"`.
* **`file.filename`*&#x2A; &#x2A;(optional)*: original filename, shown in Lounge and forwarded for context.
* **`file.file_data`**: base64-encoded data URL of the form `data:<mime-type>;base64,<data>`.

<Callout type="info">
  The `file.file_id` field (for referencing files uploaded via a provider's
  Files API) is accepted by the schema but not currently supported by the Google
  transform. Use `file_data` with an inline base64 data URL.
</Callout>

## Supported File Types [#supported-file-types]

The accepted MIME types depend on the target model. Gemini models commonly support:

* `application/pdf`
* `text/plain`
* `text/html`
* `text/css`
* `text/javascript`
* `text/csv`
* `text/markdown`
* `text/xml`

If the upstream provider rejects the MIME type, the gateway surfaces a `400` error including the unsupported MIME type and the provider it was sent to. To use a different file type, encode the file with the matching MIME type in the data URL prefix.

## Encoding a File as a Data URL [#encoding-a-file-as-a-data-url]

Any tool that can produce base64 output works. For example, in a shell:

```bash
DATA=$(base64 -i report.pdf | tr -d '\n')
echo "data:application/pdf;base64,$DATA"
```

Or in JavaScript:

```javascript
import { readFileSync } from "node:fs";

const buffer = readFileSync("report.pdf");
const fileData = `data:application/pdf;base64,${buffer.toString("base64")}`;
```

Then pass `fileData` as the `file.file_data` value in your request.

## Multiple Documents [#multiple-documents]

You can include multiple `file` blocks in a single message, optionally mixed with text and image content:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.1-pro-preview",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Compare these two reports." },
          {
            "type": "file",
            "file": {
              "filename": "q1.pdf",
              "file_data": "data:application/pdf;base64,JVBERi0x..."
            }
          },
          {
            "type": "file",
            "file": {
              "filename": "q2.pdf",
              "file_data": "data:application/pdf;base64,JVBERi0x..."
            }
          }
        ]
      }
    ]
  }'
```

## Error Handling [#error-handling]

The gateway returns `400` for the following document-related errors:

* The selected model does not support document input.
* The `file` block is missing both `file_data` and `file_id`.
* `file_data` is not a valid base64 data URL.
* The upstream provider rejects the document's MIME type for the selected model.


# Dynamic Routes
URL: https://docs.llmgateway.io/features/dynamic-routes


Dynamic routes let you move routing logic out of your application code and into the gateway. Instead of hardcoding a model, you define a named decision graph — conditions on the request, percentage-based traffic splits, and model targets — and invoke it by putting `dynamic/<name>` in the `model` field of any OpenAI-compatible request:

```bash
curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "x-user-tier: paid" \
  -d '{
    "model": "dynamic/support",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

The graph is evaluated on every request. Once it resolves to a model, the request flows through the same [smart routing](https://docs.llmgateway.io/features/routing) as any other request: weighted provider scoring, sticky sessions, and automatic cross-provider fallback all apply to the resolved target.

<Callout>
  Dynamic routes are available on the <b>Enterprise plan</b>. Manage them under{" "}
  <b>Project Settings → Dynamic Routes</b> in the dashboard, either in a visual
  drag-and-drop editor or as raw JSON.
</Callout>

## The route graph [#the-route-graph]

A route is a JSON document with an `entry` node id and a list of nodes. Evaluation starts at `entry` and follows branches until it reaches a `model` node (route the request) or an `end` node (reject the request with a `400`).

```json
{
	"entry": "tier",
	"nodes": [
		{
			"id": "tier",
			"type": "conditional",
			"conditions": [
				{
					"field": { "source": "header", "path": "x-user-tier" },
					"op": "eq",
					"value": "paid",
					"next": "premium"
				}
			],
			"else": "split"
		},
		{ "id": "premium", "type": "model", "model": "claude-sonnet-4-6" },
		{
			"id": "split",
			"type": "percentage",
			"splits": [
				{ "weight": 80, "next": "stable" },
				{ "weight": 20, "next": "experiment" }
			]
		},
		{
			"id": "stable",
			"type": "model",
			"model": "gpt-5-nano",
			"providers": ["openai"]
		},
		{ "id": "experiment", "type": "model", "model": "gemini-2.5-flash" }
	]
}
```

Node ids may contain letters, digits, hyphens, and underscores.

## Node types [#node-types]

### `conditional` [#conditional]

Evaluates its conditions top to bottom; the first match wins and the request follows that condition's `next`. When nothing matches, the request follows `else`.

Each condition reads one field from the request:

| `field.source` | What `field.path` means                                                           |
| -------------- | --------------------------------------------------------------------------------- |
| `header`       | A request header name (case-insensitive), e.g. `x-user-tier`                      |
| `body`         | A dot-path into the JSON request body, e.g. `metadata.segment` or `max_tokens`    |
| `metadata`     | A gateway-provided request attribute: `orgId`, `projectId`, `apiKeyId`, or `plan` |

Supported operators:

| Operator   | Matches when                                                              | Value type                |
| ---------- | ------------------------------------------------------------------------- | ------------------------- |
| `eq`       | The field equals the value (compared as strings)                          | string / number / boolean |
| `neq`      | The field differs from the value — also matches when the field is missing | string / number / boolean |
| `in`       | The field equals any entry in the value array                             | array of strings          |
| `contains` | The field's string form contains the value                                | string                    |
| `gt`, `lt` | The field is numerically greater / less than the value                    | number                    |
| `exists`   | The field is present (no value)                                           | —                         |

### `percentage` [#percentage]

Splits traffic across branches by relative weight. The draw is **deterministic per session**: the split key is the request's session id (`x-session-id` and the other [sticky session routing](https://docs.llmgateway.io/features/routing#sticky-session-routing) signals), so a conversation keeps its assignment across requests instead of flip-flopping between experiment arms. Requests without a session id get an independent draw per request.

Weights are relative — `{ 80, 20 }` and `{ 4, 1 }` produce the same split.

### `model` [#model]

Terminates evaluation and routes the request to a catalog model (see the [models page](https://llmgateway.io/models) for available ids). Optional `providers` restricts routing to those providers and doubles as the ordered fallback preference; when omitted, every provider serving the model is a candidate and weighted smart routing picks the best one.

### `end` [#end]

Terminates evaluation and rejects the request with a `400`. Useful as an explicit deny branch — for example, refusing traffic that doesn't carry a required header.

## Validation [#validation]

Graphs are validated when you save a draft and again when you publish:

* every referenced node must exist and be reachable from `entry`
* cycles are rejected — evaluation is deterministic per request, so a revisited node would loop forever
* model ids and provider ids must exist in the catalog, and each listed provider must actually serve the model
* operator/value mismatches (e.g. `gt` with a non-numeric value) are rejected

An invalid graph can never be saved or published, and publishing re-validates against the live model catalog so a stale draft can't resurrect a removed model.

## Versions and rollback [#versions-and-rollback]

Edits always go to the route's **draft**. Publishing snapshots the draft as an immutable, numbered version and points the route at it; the published version is what serves traffic. Rolling back re-points the route at any previous version instantly — no data migration, no re-deploy.

A route only serves requests when it is **enabled** and has a **published version**; otherwise requests using it are rejected with a `404`.

## Observability [#observability]

Each request served through a dynamic route records the route name, published version, and the node path the evaluation took in its routing metadata, alongside the usual provider scoring details — visible in the activity log detail view.

## Reliability [#reliability]

Published routes are cached in the gateway and carry a stale fallback: when the database is temporarily unreachable, recently used routes keep resolving from cache, the same way API keys and project settings do.


# Embeddable Payments
URL: https://docs.llmgateway.io/features/embeddable-payments


The **Payments SDK** lets you drop **end-user payments + in-app credit purchases** into your product the same way Stripe Elements lets you drop in payments. Your end-users get their **own wallet**, buy credits **inside your app**, and pay per request for any model the gateway supports. LLM Gateway is the merchant of record; you set a markup and keep the margin.

<Callout type="warn">
  **Preview — opt-in only.** Embeddable Payments is in preview and enabled on an
  opt-in basis per project. The **Settings → Payments SDK** dashboard page shows
  a read-only preview until the feature is turned on for your project. [Contact
  us](mailto:contact@llmgateway.io) to request access before integrating.
</Callout>

## When to use this [#when-to-use-this]

Reach for the Payments SDK when **you embed LLM Gateway's billing on your own site** and want **your end-users to pay for the AI they use**, each with their own wallet and credit balance. It handles end-user **payments and sessions**: per-user wallets, in-app credit top-ups (Stripe), short-lived browser sessions scoped to a single wallet, and developer-margin payouts.

<Callout type="info">
  **This is a payments product, not an AI client SDK.** Do not confuse it with a
  normal AI SDK such as the OpenAI SDK or the Vercel AI SDK. If you just want to
  call models from your own backend with a single API key, use the
  [OpenAI-compatible API](https://docs.llmgateway.io/quick-start) instead — you do not need the Payments
  SDK. Use the Payments SDK only when you need to **charge your own end-users**
  and give each of them a wallet and checkout inside your app.
</Callout>

It ships as three packages:

| Package                | Runs in                   | Use it for                                                                           |
| ---------------------- | ------------------------- | ------------------------------------------------------------------------------------ |
| `@llmgateway/server`   | Your backend (secret key) | Mint end-user sessions, manage wallets/customers, verify webhooks, trigger payouts   |
| `@llmgateway/client`   | Browser (headless)        | Framework-agnostic chat/image/embeddings + balance/top-up, with auto session refresh |
| `@llmgateway/elements` | React                     | Drop-in `<Chat/>`, `<BuyCredits/>`, `<CreditBalance/>` + hooks                       |

<Callout type="info">
  A complete, runnable Next.js example lives in the templates repo:
  [**Embeddable Payments
  template**](https://github.com/theopenco/llmgateway-templates/tree/main/templates/embeddable-credits).
</Callout>

## How it works [#how-it-works]

```
Your backend ──(secret key sk_)──▶ POST /v1/sessions ──▶ ephemeral session token (es_, ~15 min)
      │                                                          │
      └────────── returns es_ to your frontend ◀────────────────┘
                                  │
        Browser (es_ + pk_) ──▶ chat / images / embeddings  ──▶ debits the end-user wallet
                            └──▶ buy credits (Stripe Elements) ─▶ credits land in the wallet
```

* Your **secret key** (`sk_…`) never leaves your backend. It mints short-lived **ephemeral session tokens** (`es_…`) scoped to one end-user wallet.
* The **browser** only ever holds the `es_…` token (and a publishable Stripe key). It calls the gateway directly; usage is billed to that user's wallet.
* **Markup is applied at top-up time**: if you set a 20% markup and a user buys $10, their wallet is credited the net spend power and your **margin accrues to your organization** for later payout.
* **Top-up bonus (optional)**: set a bonus percent to credit end-users *more* than they pay — e.g. a 50% bonus turns a $10 top-up into $15 of spend power. The extra credits are funded from **your organization's credit balance** at top-up time (capped at your available credits), so it's a promotional lever you can switch on or off anytime. Markup and bonus are independent: the bonus is applied on top of the net credited amount.

## Set up in the dashboard [#set-up-in-the-dashboard]

Before you write any code, configure the project you want to embed:

1. Open the LLM Gateway dashboard and select your project.
2. Go to **Settings → Payments SDK** and turn on **End-user sessions** (this requires the preview to be enabled for your project).
3. *(Optional)* Set a **markup percent** — the margin you earn on every top-up — and/or a **top-up bonus percent** to gift end-users extra credit (funded from your organization's credit balance).
4. Add the browser origins allowed to call the gateway, one per line (e.g. `https://app.example.com`), then click **Save Settings**.
5. Under **Platform Secret Keys**, click **Create Live Key** (or **Create Test Key**) and copy the `sk_…` value immediately.
6. Store it as a server-side environment variable, for example `LLMGATEWAY_SECRET_KEY`.

The platform secret key (`sk_…`) is different from a regular gateway API key (`llmgtwy_…`): it mints end-user sessions and must only ever be used from your backend.

<Callout>
  **Test mode.** A `sk_test_…` key is a sandbox key: end-user wallet top-ups go
  through Stripe's sandbox (use Stripe [test cards](https://docs.stripe.com/testing),
  no real charges), and its wallets are fully segregated from live ones — the same
  end-user gets independent test and live wallets. To keep sandbox money from
  buying real inference, **test-mode wallets can only call free models**: use the
  `auto` route (it picks a free model automatically) or a free model id; paid
  models return a `403`. Pair a test secret key on your backend with
  `mode="test"` on `<LLMGatewayProvider>` (see below) — the two must match.
</Callout>

<Callout type="warn">
  The platform secret key is shown only once. Do not put it in frontend code,
  browser bundles, mobile apps, or public repos. Only a keyed HMAC-SHA-256
  fingerprint and masked preview are stored, so create a replacement if you lose
  the secret.
</Callout>

## 1. Install [#1-install]

```bash
# backend
npm install @llmgateway/server
# frontend (pick one)
npm install @llmgateway/elements   # React drop-in components
npm install @llmgateway/client     # headless / non-React
```

## 2. Mint a session on your backend [#2-mint-a-session-on-your-backend]

Identify your signed-in user and mint a session bound to their wallet. Scope which models they may call.

```ts
// app/api/llmgateway/session/route.ts  (Next.js Route Handler)
import { LLMGateway } from "@llmgateway/server";

const lg = new LLMGateway({ secretKey: process.env.LLMGATEWAY_SECRET_KEY! });

export async function POST() {
	const session = await lg.sessions.create({
		customer: { externalId: "user_123" }, // your stable user id
		scope: { models: ["openai/gpt-4o-mini"] }, // lock down what they can call
		ttlSeconds: 900, // optional, default 15 min
	});
	return Response.json(session); // { sessionToken, walletId, endCustomerId, expiresAt, publishableKey }
}
```

<Callout type="warn">
  Always mint sessions server-side. Never ship your `sk_…` secret key to the
  browser.
</Callout>

## 3a. Drop in the React components [#3a-drop-in-the-react-components]

Wrap your UI in `<LLMGatewayProvider>` and use the components. `fetchSession` is how the client refreshes the short-lived token before it expires.

```tsx
"use client";
import {
	LLMGatewayProvider,
	Chat,
	CreditBalance,
	BuyCredits,
} from "@llmgateway/elements";

const fetchSession = () =>
	fetch("/api/llmgateway/session", { method: "POST" }).then((r) => r.json());

export default function Assistant({ session }) {
	return (
		<LLMGatewayProvider
			session={session}
			fetchSession={fetchSession}
			mode={process.env.NODE_ENV === "production" ? "prod" : "test"}
			appearance={{ theme: "light" }}
		>
			<CreditBalance label="Your balance" />
			<BuyCredits amount={10} />
			<Chat model="openai/gpt-4o-mini" />
		</LLMGatewayProvider>
	);
}
```

Need full control over rendering? Use the hooks instead of the components:

* `useBalance()` → `{ balance, currency, recentLedger, loading, error, refetch, refetchUntilChange }`
* `useChat({ model })` → `{ turns, send, streaming, ... }`

<Callout type="info">
  `useBalance().refetchUntilChange()` polls until the balance actually changes —
  use it after a purchase, since the wallet is credited asynchronously once the
  Stripe webhook lands.
</Callout>

## 3b. Or go headless (any framework) [#3b-or-go-headless-any-framework]

```ts
import { LLMGatewayClient } from "@llmgateway/client";

const client = new LLMGatewayClient({
	session: { token: session.sessionToken, expiresAt: session.expiresAt },
	refresh: fetchSession, // auto-refreshes ~60s before expiry
});

// stream a completion (billed to the user's wallet)
for await (const delta of client.stream({
	model: "openai/gpt-4o-mini",
	messages: [{ role: "user", content: "Hello!" }],
})) {
	process.stdout.write(delta);
}

const { balance } = await client.getBalance();
```

The headless client also exposes `chat()`, `image()`, `embeddings()`, `getBalance()`, `createTopUp(amount)`, and `getConfig()`.

### Spend limits [#spend-limits]

`getBalance()` (and `useBalance()`) also returns a `limits` object describing the spend limits enforced on the session, the amount consumed so far, and — when a windowed limit is configured — when it resets. This lets you show the user how much of their allowance is left before a request is rejected.

```ts
const { balance, limits } = await client.getBalance();

// limits: {
//   usageLimit,                // lifetime spend cap (null = uncapped)
//   usage,                     // consumed over the session's lifetime
//   periodUsageLimit,          // per-window cap (null = no windowed limit)
//   periodUsageDurationValue,  // e.g. 1
//   periodUsageDurationUnit,   // "hour" | "day" | "week" | "month"
//   currentPeriodUsage,        // consumed in the current window
//   currentPeriodStartedAt,    // ISO timestamp, or null
//   currentPeriodResetAt,      // ISO timestamp the window resets, or null
// }
```

`null` limit fields mean that cap is not configured. `currentPeriodUsage` is `"0"` and `currentPeriodResetAt` is `null` until the first spend in a fresh window.

## Buying credits [#buying-credits]

`<BuyCredits amount={10} />` creates a Stripe PaymentIntent scoped to the user's wallet, renders Stripe's `PaymentElement`, and confirms the payment. Once LLM Gateway's webhook processes it, the wallet is credited the **net** amount (after your markup) and your margin accrues to your organization. If you've configured a **top-up bonus**, the additional developer-funded credit is applied on top of the net amount at the same time (debited from your organization's credit balance). The top-up API response and the `wallet.credited` webhook include `bonusCredited` and `totalCredited` so you can show the boosted total.

`@llmgateway/elements` bundles LLM Gateway's browser-safe Stripe publishable keys. Pass `mode="test"` to `<LLMGatewayProvider>` while developing to use Stripe test mode; omit it or pass `mode="prod"` for live payments (`"prod"` is the default). You never need to provide LLM Gateway's Stripe publishable key yourself, and the end-user never sees your `sk_…` secret key.

<Callout type="warn">
  The frontend `mode` prop and the backend secret key must match. A `sk_test_…`
  key creates the top-up PaymentIntent in the Stripe sandbox, which only the
  `mode="test"` publishable key can confirm — mixing a test key with `mode="prod"`
  (or vice versa) makes `<BuyCredits>` fail to confirm.
</Callout>

## Managing wallets & customers (server-side) [#managing-wallets--customers-server-side]

```ts
// grant credits directly (e.g. free trial)
await lg.wallets.credit({ walletId, amount: 5, reason: "Signup bonus" });

const wallet = await lg.wallets.retrieve(walletId);

// analytics: customers with balances + lifetime spend
const { customers } = await lg.customers.list();
const detail = await lg.customers.retrieve(endCustomerId);
```

## Webhooks [#webhooks]

Register an endpoint to react to wallet events. Events are signed (`X-LLMGateway-Signature`); verify them like Stripe.

```ts
await lg.webhookEndpoints.create({
	url: "https://yourapp.com/webhooks/llmgateway",
	enabledEvents: ["wallet.credited", "wallet.low_balance"],
});

// in your handler
const event = lg.webhooks.constructEvent(
	rawBody,
	signatureHeader,
	endpointSecret,
);
```

<Callout type="warn">
  Webhook URLs must be **https** and public — requests to private/internal
  addresses are rejected (SSRF protection), both at registration and at delivery
  time.
</Callout>

## Margin payouts (Stripe Connect) [#margin-payouts-stripe-connect]

Your accrued markup is held as a margin balance. Onboard a connected account and pay it out:

```ts
const { url } = await lg.connect.createOnboardingLink({
	refreshUrl: "https://yourapp.com/settings/payouts",
	returnUrl: "https://yourapp.com/settings/payouts?done=1",
});
// redirect the developer to `url`, then later:
const status = await lg.connect.status(); // { onboarded, payoutsEnabled, marginBalance }
const payout = await lg.connect.payout(); // transfer the accrued margin out
```

## Security model [#security-model]

* **Ephemeral tokens** (`es_…`) are short-lived and revocable; mint them per-user from your backend. Platform secrets and ephemeral tokens are stored only as keyed fingerprints, never as retrievable plaintext.
* **Model scopes** restrict each session to an allow-list of models.
* **Origin allowlist** (configured on the project) blocks browser calls from unexpected origins.
* **Per-session spend caps** (`scope.maxSpend`) bound how much a single session can spend.

## Full example [#full-example]

The end-to-end Next.js app — backend session route, provider, chat, and buy-credits — is in the templates repo:

➡️ [**Embeddable Payments template**](https://github.com/theopenco/llmgateway-templates/tree/main/templates/embeddable-credits)


# Embeddings
URL: https://docs.llmgateway.io/features/embeddings


LLMGateway exposes an OpenAI-compatible `/v1/embeddings` endpoint for generating vector representations of text — useful for semantic search, clustering, recommendations, and RAG.

Browse available embedding models on the [models page](https://llmgateway.io/models?filters=1\&embedding=true).

## Supported providers [#supported-providers]

* **OpenAI** — `text-embedding-3-small`, `text-embedding-3-large`, `text-embedding-ada-002`
* **Google AI Studio** — `gemini-embedding-2` (recommended), `gemini-embedding-001` (legacy)
* **Google Vertex AI** — `gemini-embedding-001`, `text-embedding-005`

The gateway translates between provider-native request/response shapes (e.g. Google's `:embedContent` / `:batchEmbedContents`) and the OpenAI-compatible payload, so you can swap models without changing your client code.

## cURL [#curl]

```bash
curl -X POST "https://api.llmgateway.io/v1/embeddings" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "The quick brown fox jumps over the lazy dog."
  }'
```

## OpenAI JS SDK [#openai-js-sdk]

```ts
import OpenAI from "openai";

const client = new OpenAI({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
	baseURL: "https://api.llmgateway.io/v1",
});

const response = await client.embeddings.create({
	model: "text-embedding-3-small",
	input: "The quick brown fox jumps over the lazy dog.",
});

console.log(response.data[0].embedding);
```

<Callout type="info">
  Embedding models are billed only for input tokens. There are no output tokens
  since embeddings are fixed-size vectors.
</Callout>

## Improving retrieval quality [#improving-retrieval-quality]

Vector similarity is fast but approximate. For RAG, pair embeddings with
[rerank](https://docs.llmgateway.io/features/rerank): use embeddings to pull a broad candidate set, then
rerank those candidates to pick the few documents that actually go in the
prompt.


# Guardrails
URL: https://docs.llmgateway.io/features/guardrails


Guardrails protect your organization by automatically detecting and blocking harmful content in LLM requests before they reach the model.

<Callout type="info">
  Guardrails are available on the [**Enterprise
  plan**](https://llmgateway.io/enterprise).
</Callout>

## Overview [#overview]

Guardrails run on every API request, scanning message content for:

* Security threats (prompt injection, jailbreak attempts)
* Sensitive data (PII, secrets, credentials)
* Policy violations (blocked terms, restricted topics)

When a violation is detected, you control what happens: block the request, redact the content, or log a warning.

## Organization and Project Scopes [#organization-and-project-scopes]

Guardrails are configured for the whole organization by default, and every project uses that configuration.

A project can opt out and define its own guardrails instead — useful when one application needs stricter or looser rules than the rest of the organization. Turn off **Use organization guardrails** in the project's Guardrails settings, and that project's configuration and custom rules fully replace the organization's for its traffic.

Exactly one scope is ever in force for a request:

| Project setting                     | What runs on that project's requests                      |
| ----------------------------------- | --------------------------------------------------------- |
| **Use organization guardrails** on  | The organization configuration and its custom rules       |
| **Use organization guardrails** off | The project's own configuration and its custom rules only |

Opting out never inherits partially: the organization's custom rules stop applying to that project too, and the project starts with a copy of the organization's current settings so nothing is silently dropped at the moment you opt out. Other projects are unaffected either way.

Both pages state which scope wins. A project page names itself and, while it inherits, shows the organization settings read-only; the organization page lists any projects that override it, so it is clear those settings apply everywhere else.

Only organization owners and admins can change guardrails, at either scope.

## System Rules [#system-rules]

Built-in rules protect against common threats:

### Prompt Injection Detection [#prompt-injection-detection]

Detects attempts to override or manipulate system instructions. Common patterns include:

* "Ignore all previous instructions"
* "You are now a different AI"
* Hidden instructions in encoded text

### Jailbreak Detection [#jailbreak-detection]

Identifies attempts to bypass safety measures:

* DAN (Do Anything Now) prompts
* Roleplay-based bypasses
* Instruction override attempts

### PII Detection [#pii-detection]

Identifies personal information:

* Email addresses
* Phone numbers
* Social Security Numbers
* Credit card numbers
* IP addresses
* Passport and driver's license numbers

When the action is set to **redact**, PII is replaced with placeholders like `[EMAIL_REDACTED]`.

Detection is validated rather than purely pattern-based, so identifiers that only
look like PII are left alone: card numbers must pass the Luhn checksum, phone
numbers need grouping separators or a nearby phone keyword, and passport and
license numbers need a nearby keyword. A bare numeric id such as a product id,
order number or unix timestamp is therefore never redacted as a phone number.

### Secrets Detection [#secrets-detection]

Detects credentials and API keys:

* AWS access keys and secrets
* Generic API keys
* Passwords in common formats
* Private keys

High-entropy checks and placeholder filtering keep ordinary content out of the
way: git SHAs and hex digests are not treated as AWS secrets, and template
values such as `YOUR_API_KEY`, `${OPENAI_API_KEY}` or `********` are not treated
as credentials.

### File Type Restrictions [#file-type-restrictions]

Control which file types can be uploaded:

* Configure allowed MIME types
* Set maximum file size limits
* Block potentially dangerous file types

### Document Leakage Prevention [#document-leakage-prevention]

Detects attempts to extract confidential documents or internal data.

## Configurable Actions [#configurable-actions]

For each rule, choose how to respond:

| Action     | Behavior                                            |
| ---------- | --------------------------------------------------- |
| **Block**  | Reject the request with a content policy error      |
| **Redact** | Remove or mask the sensitive content, then continue |
| **Warn**   | Log the violation but allow the request to proceed  |

## Custom Rules [#custom-rules]

Create custom rules for your use case, on the organization or on a single project:

### Blocked Terms [#blocked-terms]

Prevent specific words or phrases from being used:

* Match type: exact, contains, or regex
* Case-sensitive matching option
* Multiple terms per rule

### Custom Regex [#custom-regex]

Match patterns unique to your organization:

* Internal project codenames
* Customer identifiers
* Domain-specific sensitive data

### Topic Restrictions [#topic-restrictions]

Block content related to specific topics:

* Define restricted topics
* Keyword-based detection

## Security Events Dashboard [#security-events-dashboard]

Monitor all guardrail violations with a dedicated dashboard:

* **Total violations** — Overall count and trends
* **By action** — Breakdown of blocked, redacted, and warned
* **By category** — Which rules are being triggered
* **Detailed logs** — Individual violations with timestamps and matched patterns

## How It Works [#how-it-works]

```
Request → Guardrails Check → Action Based on Rules → Forward to Model (if allowed)
                ↓
           Log Violation
```

1. **Request received** — API request comes in with messages
2. **Scope resolved** — The project's own guardrails if it overrides, otherwise the organization's
3. **Content scanned** — All text content is checked against enabled rules
4. **Violations detected** — Matches are identified and logged
5. **Action taken** — Based on rule configuration (block/redact/warn)
6. **Request proceeds** — If not blocked, the (potentially redacted) request continues

## Best Practices [#best-practices]

1. **Start with warnings** — Enable rules in warn mode first to understand your traffic patterns
2. **Review violations** — Check the Security Events dashboard regularly
3. **Tune custom rules** — Adjust blocked terms and regex patterns based on false positives
4. **Layer defenses** — Use multiple rule types together for comprehensive protection

## Get Started [#get-started]

Guardrails are an Enterprise feature. [Contact us](https://llmgateway.io/enterprise) to enable Enterprise for your organization.


# Image Generation
URL: https://docs.llmgateway.io/features/image-generation


LLMGateway supports image generation through two APIs:

1. **`/v1/images/generations`** — OpenAI-compatible images endpoint (recommended for simple image generation)
2. **`/v1/images/edits`** — OpenAI-compatible image editing endpoint
3. **`/v1/chat/completions`** — Chat completions with image generation models (for conversational image generation and editing)

For asynchronous video generation, see [Video Generation](https://docs.llmgateway.io/features/video-generation).

## Available Models [#available-models]

You can find all available image generation models on our [models page](https://llmgateway.io/models?filters=1\&imageGeneration=true).

## OpenAI Images API [#openai-images-api]

The `/v1/images/generations` endpoint provides a drop-in replacement for OpenAI's image generation API. It works with any OpenAI-compatible client library.

### Parameters [#parameters]

| Parameter         | Type    | Default      | Description                                                                                                      |
| ----------------- | ------- | ------------ | ---------------------------------------------------------------------------------------------------------------- |
| `prompt`          | string  | required     | A text description of the desired image(s)                                                                       |
| `model`           | string  | `"auto"`     | The model to use. `auto` resolves to `gemini-3-pro-image`                                                        |
| `n`               | integer | `1`          | Number of images to generate (1-10)                                                                              |
| `size`            | string  | —            | Image dimensions. Supported sizes depend on the model/provider — see [Image Configuration](#image-configuration) |
| `quality`         | string  | —            | Image quality. Supported values depend on the model/provider — see [Image Configuration](#image-configuration)   |
| `response_format` | string  | `"b64_json"` | Only `b64_json` is supported                                                                                     |
| `style`           | string  | —            | Image style: `vivid` or `natural`                                                                                |

### curl [#curl]

```bash
curl -X POST "https://api.llmgateway.io/v1/images/generations" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3-pro-image",
    "prompt": "A cute cat wearing a tiny top hat",
    "n": 1,
    "size": "1024x1024"
  }'
```

### OpenAI SDK [#openai-sdk]

Works with the standard OpenAI client library — just point the base URL to LLMGateway.

```ts
import OpenAI from "openai";
import { writeFileSync } from "fs";

const client = new OpenAI({
	baseURL: "https://api.llmgateway.io/v1",
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const response = await client.images.generate({
	model: "gemini-3-pro-image",
	prompt: "A futuristic city skyline at sunset with flying cars",
	n: 1,
	size: "1024x1024",
});

response.data.forEach((image, i) => {
	if (image.b64_json) {
		const buf = Buffer.from(image.b64_json, "base64");
		writeFileSync(`image-${i}.png`, buf);
	}
});
```

### Vercel AI SDK [#vercel-ai-sdk]

Use the `@llmgateway/ai-sdk-provider` with `generateImage`.

```ts
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { generateImage } from "ai";
import { writeFileSync } from "fs";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const result = await generateImage({
	model: llmgateway.image("gemini-3-pro-image"),
	prompt:
		"A cozy cabin in a snowy mountain landscape at night with aurora borealis",
	size: "1024x1024",
	n: 1,
	// aspectRatio and quality are model-specific — only some providers honor them.
	// aspectRatio works on Gemini image models; OpenAI gpt-image-2 ignores it
	// (use a literal WxH `size` instead).
	aspectRatio: "16:9",
	// quality works on OpenAI gpt-image-2 ("low" | "medium" | "high" | "auto").
	// The AI SDK only forwards it through providerOptions.
	providerOptions: {
		llmgateway: { quality: "high" },
	},
});

result.images.forEach((image, i) => {
	const buf = Buffer.from(image.base64, "base64");
	writeFileSync(`image-${i}.png`, buf);
});
```

## OpenAI Images Edit API [#openai-images-edit-api]

The `/v1/images/edits` endpoint is OpenAI-compatible and supports a focused subset of `images.edit` parameters.

### Parameters [#parameters-1]

| Parameter            | Type                     | Required | Description                                                        |
| -------------------- | ------------------------ | -------- | ------------------------------------------------------------------ |
| `images`             | array of `{ image_url }` | yes      | Input images. `image_url` supports HTTPS URLs and base64 data URLs |
| `prompt`             | string                   | yes      | A text description of the desired image edit                       |
| `model`              | string                   | no       | Image editing model                                                |
| `background`         | enum                     | no       | `transparent`, `opaque`, or `auto`                                 |
| `input_fidelity`     | enum                     | no       | `high` or `low`                                                    |
| `n`                  | integer                  | no       | Number of edited images to generate                                |
| `output_format`      | enum                     | no       | `png`, `jpeg`, or `webp`                                           |
| `output_compression` | integer                  | no       | Compression level for `jpeg`/`webp`                                |
| `quality`            | enum                     | no       | `low`, `medium`, `high`, or `auto`                                 |
| `size`               | string                   | no       | Output size. Examples: `1024x1024`, `1536x1024`, `1K`, `2K`, `4K`  |
| `aspect_ratio`       | string                   | no       | Aspect ratio override. Examples: `1:1`, `16:9`, `4:3`, `5:4`       |

<Callout type="warning">
  `mask` is not supported yet on `/v1/images/edits`.
</Callout>

### curl (HTTPS image URL) [#curl-https-image-url]

```bash
curl -X POST "https://api.llmgateway.io/v1/images/edits" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "images": [
      {
        "image_url": "https://example.com/source-image.png"
      }
    ],
    "prompt": "Add a watercolor effect to this image",
    "model": "gemini-3-pro-image",
    "aspect_ratio": "16:9",
    "quality": "high",
    "size": "4K"
  }'
```

### curl (base64 data URL) [#curl-base64-data-url]

```bash
curl -X POST "https://api.llmgateway.io/v1/images/edits" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "images": [
      {
        "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
      }
    ],
    "prompt": "Turn this into a pixel-art style image"
  }'
```

## Chat Completions API [#chat-completions-api]

Image generation also works through the `/v1/chat/completions` endpoint, which is useful for conversational image generation, image editing with vision, and multi-turn interactions.

### Making Requests [#making-requests]

Simply use an image generation model and provide a text prompt describing the image you want to create.

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3-pro-image",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a cute golden retriever puppy playing in a sunny meadow"
      }
    ]
  }'
```

### Response Format [#response-format]

Image generation models return responses in the standard chat completions format, with generated images included in the `images` array within the assistant message:

```json
{
	"id": "chatcmpl-1756234109285",
	"object": "chat.completion",
	"created": 1756234109,
	"model": "gemini-3-pro-image",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "Here's an image of a cute dog for you: ",
				"images": [
					{
						"type": "image_url",
						"image_url": {
							"url": "data:image/png;base64,<base64_encoded_image_data>"
						}
					}
				]
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 8,
		"completion_tokens": 1303,
		"total_tokens": 1311
	}
}
```

### Vision support [#vision-support]

You can edit or modify images by combining image generation with [vision models](https://docs.llmgateway.io/features/vision) by including the image in the `messages` array.

### Response Structure [#response-structure]

#### Images Array [#images-array]

The `images` array contains one or more generated images with the following structure:

* `type`: Always `"image_url"` for generated images
* `image_url.url`: A data URL containing the base64-encoded image data (format: `data:image/png;base64,<data>`)

#### Content Field [#content-field]

The `content` field may contain descriptive text about the generated image, depending on the model's behavior.

### AI SDK (Chat Completions) [#ai-sdk-chat-completions]

You can use the AI SDK to generate images with your existing generateText or streamText calls using the LLMGateway provider.

#### Example [#example]

```ts title="/api/chat/route.ts"
import { streamText, type UIMessage, convertToModelMessages } from "ai";
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";

interface ChatRequestBody {
	messages: UIMessage[];
}

export async function POST(req: Request) {
	const body = await req.json();

	const { messages }: ChatRequestBody = body;

	const llmgateway = createLLMGateway({
		apiKey: "llmgateway_api_key",
		baseUrl: "https://api.llmgateway.io/v1",
	});

	try {
		const result = streamText({
			model: llmgateway.chat("gemini-3-pro-image"),
			messages: convertToModelMessages(messages),
		});

		return result.toUIMessageStreamResponse();
	} catch {
		return new Response(
			JSON.stringify({ error: "LLM Gateway request failed" }),
			{
				status: 500,
			},
		);
	}
}
```

Then you can render the image in your frontend using the `Image` component from the [ai-elements](https://ai-sdk.dev/elements/components/image).

Here is a full example of how to use the AI SDK to generate images in your frontend:

```tsx title="/app/page.tsx"
"use client";

import { useState, useRef } from "react";
import { useChat } from "@ai-sdk/react";
import { parseImagePartToDataUrl } from "@/lib/image-utils";
import {
	PromptInput,
	PromptInputBody,
	PromptInputButton,
	PromptInputSubmit,
	PromptInputTextarea,
	PromptInputToolbar,
} from "@/components/ai-elements/prompt-input";
import {
	Conversation,
	ConversationContent,
} from "@/components/ai-elements/conversation";
import { Image } from "@/components/ai-elements/image";
import { Loader } from "@/components/ai-elements/loader";
import { Message, MessageContent } from "@/components/ai-elements/message";
import { Response } from "@/components/ai-elements/response";

export const ChatUI = () => {
	const textareaRef = useRef<HTMLTextAreaElement | null>(null);
	const [text, setText] = useState("");
	const { messages, status, stop, regenerate, sendMessage } = useChat();

	return (
		<>
			<div className="flex-1 overflow-y-auto px-4 pb-24">
				<Conversation>
					<ConversationContent>
						{messages.length === 0 ? (
							<div className="mb-6 text-center">
								<h2 className="text-3xl font-semibold tracking-tight">
									How can I help you?
								</h2>
							</div>
						) : (
							messages.map((m, messageIndex) => {
								const isLastMessage = messageIndex === messages.length - 1;

								if (m.role === "assistant") {
									const textContent = m.parts
										.filter((p) => p.type === "text")
										.map((p) => p.text)
										.join("");
									// Combine all image parts (both image_url and file types)
									const imageParts = m.parts.filter(
										(p) =>
											p.type === "file" && p.mediaType?.startsWith("image/"),
									);

									return (
										<div key={m.id}>
											{textContent ? <Response>{textContent}</Response> : null}
											{imageParts.length > 0 ? (
												<div className="mt-3 grid grid-cols-1 gap-3 sm:grid-cols-2">
													{imageParts.map((part, idx: number) => {
														const { base64Only, mediaType } =
															parseImagePartToDataUrl(part);

														if (!base64Only) {
															return null;
														}

														return (
															<Image
																key={idx}
																base64={base64Only}
																mediaType={mediaType}
																alt={part.name || "Generated image"}
															/>
														);
													})}
												</div>
											) : null}
											{isLastMessage &&
												(status === "submitted" || status === "streaming") && (
													<Loader />
												)}
										</div>
									);
								} else {
									return (
										<Message key={m.id} from={m.role}>
											<MessageContent variant="flat">
												{m.parts.map((p, i) => {
													if (p.type === "text") {
														return <div key={i}>{p.text}</div>;
													}
													return null;
												})}
											</MessageContent>
											{isLastMessage &&
												(status === "submitted" || status === "streaming") && (
													<Loader />
												)}
										</Message>
									);
								}
							})
						)}
					</ConversationContent>
				</Conversation>
			</div>
			<div className="sticky bottom-0 left-0 right-0 px-4 pb-[max(env(safe-area-inset-bottom),1rem)] pt-2 bg-gradient-to-t from-background via-background/95 to-transparent backdrop-blur supports-[backdrop-filter]:bg-background/60">
				<PromptInput
					aria-disabled={status === "streaming"}
					onSubmit={async (message) => {
						if (status === "streaming") {
							return;
						}

						try {
							const textContent = message.text ?? "";
							if (!textContent.trim()) {
								return;
							}

							setText(""); // Clear input immediately

							const parts = [{ type: "text", text: textContent }];

							// Call sendMessage which will handle adding the user message and API request
							sendMessage({
								role: "user",
								parts,
							});
						} catch (error) {
							// Throw error here
						}
					}}
				>
					<PromptInputBody>
						<PromptInputTextarea
							ref={textareaRef}
							value={text}
							onChange={(e) => setText(e.currentTarget.value)}
							placeholder="Message"
						/>
					</PromptInputBody>
					<PromptInputToolbar>
						<div className="flex items-center gap-2">
							{status === "streaming" ? (
								<PromptInputButton onClick={() => stop()} variant="ghost">
									Stop
								</PromptInputButton>
							) : null}
							<PromptInputSubmit
								status={status === "streaming" ? "streaming" : "ready"}
							/>
						</div>
					</PromptInputToolbar>
				</PromptInput>
			</div>
		</>
	);
};
```

```ts title="/lib/image-utils.ts"
/**
 * Parses a file object containing image data and returns a properly formatted data URL
 * and normalized media type.
 *
 * Handles:
 * - Normalizing mediaType from various property names (mediaType, mime_type)
 * - Detecting existing data: URLs
 * - Detecting base64-looking content
 * - Stripping whitespace from base64 content
 * - Building proper data:...;base64,... URLs
 */
export function parseImageFile(file: {
	url?: string;
	mediaType?: string;
	mime_type?: string;
}): { dataUrl: string; mediaType: string } {
	const mediaType = file.mediaType || file.mime_type || "image/png";
	let url = String(file.url || "");

	const isDataUrl = url.startsWith("data:");
	const looksLikeBase64 =
		!isDataUrl && /^[A-Za-z0-9+/=\s]+$/.test(url.slice(0, 200));

	if (looksLikeBase64) {
		url = url.replace(/\s+/g, "");
	}

	const dataUrl = isDataUrl
		? url
		: looksLikeBase64
			? `data:${mediaType};base64,${url}`
			: url;

	return { dataUrl, mediaType };
}

/**
 * Extracts base64-only content from a data URL.
 * Returns empty string if the input is not a valid data URL.
 */
export function extractBase64FromDataUrl(dataUrl: string): string {
	if (!dataUrl.startsWith("data:")) {
		return "";
	}

	const comma = dataUrl.indexOf(",");
	return comma >= 0 ? dataUrl.slice(comma + 1) : "";
}

/**
 * Parses an image part (either image_url or file type) and returns
 * dataUrl, base64Only, and mediaType ready for rendering.
 *
 * Handles error cases gracefully by returning empty base64Only string
 * when parsing fails, allowing the renderer to skip invalid images.
 */
export function parseImagePartToDataUrl(part: any): {
	dataUrl: string;
	base64Only: string;
	mediaType: string;
} {
	try {
		// Handle image_url parts
		if (part.type === "image_url" && part.image_url?.url) {
			const url = part.image_url.url;
			const mediaType = "image/png"; // Default for image_url parts

			if (url.startsWith("data:")) {
				// Extract media type from data URL if present
				const match = url.match(/data:([^;]+)/);
				const extractedMediaType = match?.[1] || mediaType;
				return {
					dataUrl: url,
					base64Only: extractBase64FromDataUrl(url),
					mediaType: extractedMediaType,
				};
			}

			return {
				dataUrl: url,
				base64Only: "",
				mediaType,
			};
		}

		// Handle file parts (AI SDK format)
		if (part.type === "file") {
			const { dataUrl, mediaType } = parseImageFile(part);
			return {
				dataUrl,
				base64Only: extractBase64FromDataUrl(dataUrl),
				mediaType,
			};
		}

		return {
			dataUrl: "",
			base64Only: "",
			mediaType: "image/png",
		};
	} catch {
		return {
			dataUrl: "",
			base64Only: "",
			mediaType: "image/png",
		};
	}
}
```

## Image Configuration [#image-configuration]

You can customize the generated image using the optional `image_config` parameter (for chat completions) or `size`/`quality`/`style` parameters (for the images API). The supported parameters vary by provider.

### Google Models [#google-models]

Available Google models:

| Model                    | Description                                                                         |
| ------------------------ | ----------------------------------------------------------------------------------- |
| `gemini-3-pro-image`     | Gemini 3 Pro with native image generation. Supports aspect ratios and 1K–4K sizes.  |
| `gemini-3.1-flash-image` | Gemini 3.1 Flash with native image generation. Supports 0.5K–4K sizes (default 1K). |

#### gemini-3-pro-image [#gemini-3-pro-image]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3-pro-image",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a mountain landscape at sunset"
      }
    ],
    "image_config": {
      "aspect_ratio": "16:9",
      "image_size": "4K"
    }
  }'
```

| Parameter      | Type   | Description                                                                                                                                   |
| -------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `aspect_ratio` | string | The aspect ratio of the generated image. Options: `"1:1"`, `"2:3"`, `"3:2"`, `"3:4"`, `"4:3"`, `"4:5"`, `"5:4"`, `"9:16"`, `"16:9"`, `"21:9"` |
| `image_size`   | string | The resolution of the generated image. Options: `"1K"` (1024x1024), `"2K"` (2048x2048), `"4K"` (4096x4096)                                    |

#### gemini-3.1-flash-image [#gemini-31-flash-image]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.1-flash-image",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a mountain landscape at sunset"
      }
    ],
    "image_config": {
      "image_size": "1K"
    }
  }'
```

| Parameter      | Type   | Description                                                                                                                                                                       |
| -------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `aspect_ratio` | string | The aspect ratio of the generated image. Options: `"1:1"`, `"1:4"`, `"1:8"`, `"2:3"`, `"3:2"`, `"3:4"`, `"4:1"`, `"4:3"`, `"4:5"`, `"5:4"`, `"8:1"`, `"9:16"`, `"16:9"`, `"21:9"` |
| `image_size`   | string | The resolution of the generated image. Options: `"0.5K"` (512x512), `"1K"` (1024x1024, default), `"2K"` (2048x2048), `"4K"` (4096x4096)                                           |

<Callout type="info">
  `gemini-3.1-flash-image` uniquely supports `"0.5K"` resolution, which is not
  available on other Google image models.
</Callout>

### Meta Models [#meta-models]

Muse Image uses Meta's Responses API for generation and editing. It reasons
before rendering and can use reference images across refinement turns.

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/muse-image-1.0",
    "messages": [
      {
        "role": "user",
        "content": "Create a product photo on a warm studio background"
      }
    ],
    "image_config": {
      "image_size": "1024x1536"
    }
  }'
```

| Parameter    | Type   | Description                                                                |
| ------------ | ------ | -------------------------------------------------------------------------- |
| `image_size` | string | One of `"1024x1024"`, `"1024x1536"`, or `"1536x1024"`. Defaults to square. |

Muse Image does not expose a quality setting. Use `image_size` to choose square,
portrait, or landscape output.

### Alibaba Models [#alibaba-models]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "alibaba/qwen-image-plus",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a mountain landscape at sunset"
      }
    ],
    "image_config": {
      "image_size": "1024x1536",
      "n": 1,
      "seed": 42
    }
  }'
```

| Parameter    | Type    | Description                                                                                      |
| ------------ | ------- | ------------------------------------------------------------------------------------------------ |
| `image_size` | string  | Image dimensions in `WIDTHxHEIGHT` format. Examples: `"1024x1024"`, `"1024x1536"`, `"1536x1024"` |
| `n`          | integer | Number of images to generate (1-4)                                                               |
| `seed`       | integer | Random seed for reproducible generation                                                          |

Available Alibaba models (see the [models page](https://llmgateway.io/models?filters=1\&imageGeneration=true) for current pricing):

| Model                        | Description                                                                          |
| ---------------------------- | ------------------------------------------------------------------------------------ |
| `alibaba/qwen-image`         | Standard quality image generation                                                    |
| `alibaba/qwen-image-plus`    | Good balance of quality and cost                                                     |
| `alibaba/qwen-image-max`     | Highest quality image generation                                                     |
| `alibaba/qwen-image-3.0`     | Third-generation image generation and editing                                        |
| `alibaba/qwen-image-3.0-pro` | Highest quality third-generation generation and editing. Priced per output size tier |

<Callout type="info">
  Alibaba models use explicit pixel dimensions (e.g., `"1024x1536"`) instead of
  aspect ratios. For portrait orientation use `"1024x1536"`, for landscape use
  `"1536x1024"`.
</Callout>

### Z.AI Models [#zai-models]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai/cogview-4",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a futuristic city skyline"
      }
    ],
    "image_config": {
      "image_size": "1024x1024"
    }
  }'
```

| Parameter    | Type    | Description                                                                                      |
| ------------ | ------- | ------------------------------------------------------------------------------------------------ |
| `image_size` | string  | Image dimensions in `WIDTHxHEIGHT` format. Examples: `"1024x1024"`, `"2048x1024"`, `"1024x2048"` |
| `n`          | integer | Number of images to generate                                                                     |

Available Z.AI models (see the [models page](https://llmgateway.io/models?filters=1\&imageGeneration=true) for current pricing):

| Model           | Description                                                                                                         |
| --------------- | ------------------------------------------------------------------------------------------------------------------- |
| `zai/cogview-4` | CogView-4 with bilingual support and excellent text rendering                                                       |
| `zai/glm-image` | GLM-Image with hybrid auto-regressive architecture, excellent for text-rendering and knowledge-intensive generation |

<Callout type="info">
  CogView-4 supports both Chinese and English prompts and excels at generating
  images with embedded text.
</Callout>

### OpenAI Models [#openai-models]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-image-2",
    "messages": [
      {
        "role": "user",
        "content": "Generate a photo-real cinematic landscape at golden hour"
      }
    ],
    "image_config": {
      "image_size": "3072x2160",
      "image_quality": "low"
    }
  }'
```

| Parameter       | Type   | Description                                                                           |
| --------------- | ------ | ------------------------------------------------------------------------------------- |
| `image_size`    | string | Image dimensions in `WIDTHxHEIGHT` format, or `"auto"` to let the model choose.       |
| `image_quality` | string | One of `"low"`, `"medium"`, `"high"`, or `"auto"`. Defaults to `"auto"` when omitted. |

<Callout type="warning">
  OpenAI image models do **not** accept `aspect_ratio`. Always specify
  `image_size` as `WIDTHxHEIGHT` (e.g. `"1024x1024"`, `"3072x2160"`). OpenAI
  requires both width and height to be divisible by 16, the longest edge to be ≤
  3840, and the total pixel count to fit within the model's pixel budget;
  requests outside these bounds are rejected with HTTP 400.
</Callout>

Available OpenAI image models:

| Model                | Description                                                                                                  |
| -------------------- | ------------------------------------------------------------------------------------------------------------ |
| `openai/gpt-image-2` | OpenAI's next-generation image model with improved quality and prompt adherence, supporting text and vision. |

### ByteDance Models [#bytedance-models]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bytedance/seedream-4-5",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a futuristic cyberpunk city at night"
      }
    ],
    "image_config": {
      "image_size": "2048x2048"
    }
  }'
```

| Parameter    | Type   | Description                                                                                      |
| ------------ | ------ | ------------------------------------------------------------------------------------------------ |
| `image_size` | string | Image dimensions in `WIDTHxHEIGHT` format. Examples: `"1024x1024"`, `"2048x2048"`, `"4096x4096"` |

Available ByteDance models (see the [models page](https://llmgateway.io/models?filters=1\&imageGeneration=true) for current pricing):

| Model                        | Description                                                     |
| ---------------------------- | --------------------------------------------------------------- |
| `bytedance/seedream-4-0`     | High-quality text-to-image generation with 2K default output    |
| `bytedance/seedream-4-5`     | Enhanced quality and consistency with improved prompt adherence |
| `bytedance/seedream-5-0-pro` | Precise generation and reference-image editing at 1K or 2K      |

<Callout type="info">
  Seedream models support up to 2-10 reference images for multi-image fusion and
  generation. The default output resolution is 2048×2048 (2K), with support up
  to 4096×4096 (4K).
</Callout>

## Usage Notes [#usage-notes]

<Callout type="info">
  Image generation models typically have higher token costs compared to
  text-only models due to the computational requirements of image synthesis.
</Callout>

<Callout type="warning">
  Generated images are returned as base64-encoded data URLs, which can be large.
  Consider the payload size when integrating image generation into your
  applications.
</Callout>


# Master Keys
URL: https://docs.llmgateway.io/features/master-keys


Master keys are org-scoped bearer tokens that let you create projects, gateway API keys, IAM rules (both per key and per organization member), and your organization's custom providers and custom models programmatically — without going through the dashboard. They are intended for server-to-server provisioning (e.g. multi-tenant onboarding from your own backend).

They also expose [usage and cost reporting](#usage-and-cost-reporting), so you can pull per-member and per-model spend into your own dashboards, data warehouse, or chargeback process.

<Callout type="info">
  Master keys are available on the **Enterprise** plan only. Contact us at
  [contact@llmgateway.io](mailto:contact@llmgateway.io) to enable them for your organization.
</Callout>

## Security [#security]

* Master keys are stored as **HMAC-SHA256 hashes** in the database (using the `GATEWAY_API_KEY_HASH_SECRET` secret). The plain token is shown to you **only once** at creation time.
* Each master key is scoped to a single organization and cannot access resources in other organizations.
* Deleting or deactivating a master key revokes all programmatic access immediately.
* All creates/deletes/status changes are recorded in your organization audit log.

## Limits [#limits]

* Maximum **10 active master keys per organization**.
* Programmatic project and API-key creation enforces the same per-org and per-project limits as the dashboard flow.

## Managing master keys [#managing-master-keys]

In the dashboard, go to **Organization → Master Keys**. From there you can:

* Create a new master key (the plain token is shown once — copy it immediately).
* View the masked token, status, creator, and last-used timestamp for each existing key.
* Activate / deactivate or delete keys.

## Authentication [#authentication]

All programmatic endpoints live under `/v1/master/*` and require a master key in the `Authorization` header:

```
Authorization: Bearer llmgmk_...
```

A request with a missing, invalid, inactive, or non-enterprise master key receives a 401 / 403 response.

## Endpoints [#endpoints]

### List projects [#list-projects]

`GET /v1/master/projects`

Returns all non-deleted projects in the master key's organization.

```bash
curl https://internal.llmgateway.io/v1/master/projects \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"projects": [
		{
			"id": "proj_...",
			"name": "Customer ACME",
			"organizationId": "org_...",
			"cachingEnabled": false,
			"cacheDurationSeconds": 60,
			"mode": "hybrid",
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Create a project [#create-a-project]

`POST /v1/master/projects`

```bash
curl -X POST https://internal.llmgateway.io/v1/master/projects \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Customer ACME",
    "cachingEnabled": false,
    "mode": "hybrid"
  }'
```

Body parameters:

| Field                  | Type                                             | Description                |
| ---------------------- | ------------------------------------------------ | -------------------------- |
| `name`                 | string                                           | Project name (1–255 chars) |
| `cachingEnabled`       | boolean (optional)                               | Default `false`            |
| `cacheDurationSeconds` | number (optional)                                | 10–31536000, default 60    |
| `mode`                 | `"api-keys" \| "credits" \| "hybrid"` (optional) | Default `"hybrid"`         |

Response (201): the created project.

### Update a project [#update-a-project]

`PATCH /v1/master/projects/{id}`

Updates a project owned by the master key's organization. All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.llmgateway.io/v1/master/projects/proj_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Customer ACME (renamed)",
    "cachingEnabled": true,
    "status": "inactive"
  }'
```

Body parameters (all optional, at least one required):

| Field                      | Type                                  | Description                                                       |
| -------------------------- | ------------------------------------- | ----------------------------------------------------------------- |
| `name`                     | string                                | 1–255 chars                                                       |
| `cachingEnabled`           | boolean                               |                                                                   |
| `cacheDurationSeconds`     | number                                | 10–31536000                                                       |
| `providerCacheControlMode` | `"auto" \| "passthrough" \| "off"`    | [Provider cache writes](https://docs.llmgateway.io/features/caching/provider-cache-control) |
| `mode`                     | `"api-keys" \| "credits" \| "hybrid"` |                                                                   |
| `status`                   | `"active" \| "inactive"`              | Toggle the project without deleting                               |

Response (200): the updated project. It carries both `providerCacheControlMode` and the older `providerCacheControlEnabled` boolean, which reports `false` only for `"off"`.

### Delete a project [#delete-a-project]

`DELETE /v1/master/projects/{id}`

Soft-deletes a project (sets `status` to `"deleted"`). Cascades to its API keys.

```bash
curl -X DELETE https://internal.llmgateway.io/v1/master/projects/proj_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "Project deleted successfully" }
```

### List gateway API keys [#list-gateway-api-keys]

`GET /v1/master/keys`

Returns the developer-created gateway API keys in the master key's organization, each with its creator, project, configured limits, the usage consumed so far, and — when a windowed limit is set — the time the current period resets. `projectName` provides the human-readable project name alongside `projectId`. `createdBy` contains the creator's internal user ID, while `createdByEmail` contains their email address (`null` when unavailable). Pass an optional `projectId` query parameter to scope the list to a single project.

```bash
curl "https://internal.llmgateway.io/v1/master/keys?projectId=proj_..." \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"apiKeys": [
		{
			"id": "ak_...",
			"description": "Customer ACME — production key",
			"status": "active",
			"projectId": "proj_...",
			"projectName": "Customer ACME",
			"createdBy": "usr_...",
			"createdByEmail": "member@example.com",
			"maskedToken": "llmgtwy_...abcd",
			"usageLimit": "100.00",
			"usage": "42.13",
			"periodUsageLimit": "10.00",
			"periodUsageDurationValue": 1,
			"periodUsageDurationUnit": "day",
			"currentPeriodUsage": "3.50",
			"currentPeriodStartedAt": "2025-01-15T00:00:00.000Z",
			"currentPeriodResetAt": "2025-01-16T00:00:00.000Z",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

Limit and usage fields:

| Field                      | Description                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------- |
| `usageLimit`               | Lifetime spend cap (`null` when uncapped)                                          |
| `usage`                    | Total spend accrued against `usageLimit` over the key's lifetime                   |
| `periodUsageLimit`         | Recurring per-window spend cap (`null` when no windowed limit is configured)       |
| `periodUsageDurationValue` | Length of the window, paired with `periodUsageDurationUnit`                        |
| `periodUsageDurationUnit`  | `"hour" \| "day" \| "week" \| "month"`                                             |
| `currentPeriodUsage`       | Spend accrued in the current window (`"0"` when unconfigured or the window lapsed) |
| `currentPeriodStartedAt`   | When the current window began (`null` when unconfigured or lapsed)                 |
| `currentPeriodResetAt`     | When the windowed limit resets (`null` when unconfigured or lapsed)                |

The plain token is never returned by this endpoint — only a masked form for identification.

### Get a gateway API key [#get-a-gateway-api-key]

`GET /v1/master/keys/{id}`

Returns a single gateway API key in the master key's organization, with the same limit, usage, and reset-time fields as the list endpoint.

```bash
curl https://internal.llmgateway.io/v1/master/keys/ak_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"apiKey": {
		"id": "ak_...",
		"description": "Customer ACME — production key",
		"status": "active",
		"projectId": "proj_...",
		"projectName": "Customer ACME",
		"createdBy": "usr_...",
		"createdByEmail": "member@example.com",
		"maskedToken": "llmgtwy_...abcd",
		"usageLimit": "100.00",
		"usage": "42.13",
		"periodUsageLimit": "10.00",
		"periodUsageDurationValue": 1,
		"periodUsageDurationUnit": "day",
		"currentPeriodUsage": "3.50",
		"currentPeriodStartedAt": "2025-01-15T00:00:00.000Z",
		"currentPeriodResetAt": "2025-01-16T00:00:00.000Z",
		"createdAt": "...",
		"updatedAt": "..."
	}
}
```

### Create a gateway API key [#create-a-gateway-api-key]

`POST /v1/master/keys`

```bash
curl -X POST https://internal.llmgateway.io/v1/master/keys \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "proj_...",
    "description": "Customer ACME — production key"
  }'
```

Body parameters:

| Field                      | Type                                              | Description                                  |
| -------------------------- | ------------------------------------------------- | -------------------------------------------- |
| `projectId`                | string                                            | Must belong to the master key's organization |
| `description`              | string                                            | API key description (1–255 chars)            |
| `usageLimit`               | string (optional)                                 | Lifetime usage limit                         |
| `periodUsageLimit`         | string (optional)                                 | Recurring period usage limit                 |
| `periodUsageDurationValue` | number (optional)                                 | Required if `periodUsageLimit` is set        |
| `periodUsageDurationUnit`  | `"hour" \| "day" \| "week" \| "month"` (optional) | Required if `periodUsageLimit` is set        |

<Callout type="warning">
  The created gateway API key's plain token is returned in the response **only
  once**. Persist it immediately on your side.
</Callout>

Response (201):

```json
{
	"apiKey": {
		"id": "ak_...",
		"token": "llmgtwy_...",
		"description": "Customer ACME — production key",
		"status": "active",
		"projectId": "proj_...",
		"createdBy": "usr_...",
		"createdAt": "...",
		"updatedAt": "..."
	}
}
```

### Update a gateway API key [#update-a-gateway-api-key]

`PATCH /v1/master/keys/{id}`

Updates an API key in a project owned by the master key's organization. All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.llmgateway.io/v1/master/keys/ak_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "inactive",
    "usageLimit": "100.00"
  }'
```

Body parameters (all optional, at least one required):

| Field                      | Type                                   | Description                            |
| -------------------------- | -------------------------------------- | -------------------------------------- |
| `description`              | string                                 | 1–255 chars                            |
| `status`                   | `"active" \| "inactive"`               |                                        |
| `usageLimit`               | string \| null                         | Lifetime usage limit (null to clear)   |
| `periodUsageLimit`         | string \| null                         | Recurring period limit (null to clear) |
| `periodUsageDurationValue` | number \| null                         | Required if `periodUsageLimit` is set  |
| `periodUsageDurationUnit`  | `"hour" \| "day" \| "week" \| "month"` | Required if `periodUsageLimit` is set  |

Response (200): the updated API key, including its configured limits, consumed `usage` / `currentPeriodUsage`, and the `currentPeriodResetAt` window-reset time (same fields as the [list endpoint](#list-gateway-api-keys)). The plain token is **not** included — it is only returned at creation.

<Callout type="info">
  Two dashboard-only key features have no master API equivalent yet: [expiration
  (TTL)](https://docs.llmgateway.io/features/api-keys#expiration-ttl) and [rolling a key's
  secret](https://docs.llmgateway.io/features/api-keys#rotating-rolling-api-keys). To replace a key
  programmatically, create a new one and delete the old one once your clients
  have switched over.
</Callout>

### Delete a gateway API key [#delete-a-gateway-api-key]

`DELETE /v1/master/keys/{id}`

Soft-deletes the API key (sets `status` to `"deleted"`). Any in-flight requests using the key will be rejected immediately on next auth check.

```bash
curl -X DELETE https://internal.llmgateway.io/v1/master/keys/ak_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "API key deleted successfully" }
```

<Callout type="info">
  The managed Playground API key cannot be updated or deleted via the master
  API.
</Callout>

## Usage and cost reporting [#usage-and-cost-reporting]

`GET /v1/master/usage`

Returns usage and cost for the master key's organization as a flat list of rows, grouped by any combination of member, model, provider, project, and API key, and bucketed hourly, daily, or not at all. This is the endpoint to point an internal reporting tool, data warehouse, or chargeback job at — it is the only usage surface that authenticates with a token rather than a dashboard session.

<Callout type="info">
  Like every `/v1/master/*` endpoint, this requires an **Enterprise** plan. A
  master key on any other plan receives a `403`.
</Callout>

```bash
# Cost per member for a month
curl -G https://internal.llmgateway.io/v1/master/usage \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d from=2026-07-01 -d to=2026-07-31 \
  -d granularity=total -d groupBy=user
```

Query parameters:

| Param         | Type                                                              | Default      | Description                                                                              |
| ------------- | ----------------------------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------- |
| `from`, `to`  | `YYYY-MM-DD`                                                      | last 7 days  | Inclusive window, interpreted as wall-clock days in `timezone`. Max 366 days             |
| `timezone`    | IANA timezone                                                     | `UTC`        | Timezone the day/hour buckets are labelled in                                            |
| `granularity` | `hour` \| `day` \| `total`                                        | `day`        | Time bucket. `total` collapses the window into one row per group. `hour` caps at 31 days |
| `groupBy`     | comma-separated: `user`, `model`, `provider`, `project`, `apiKey` | `user,model` | Dimensions to break down by. Pass an empty value for organization-wide totals            |
| `projectId`   | string                                                            | —            | Restrict to one project. `404` if it is not in this organization                         |
| `userId`      | string                                                            | —            | Restrict to one member                                                                   |
| `apiKeyId`    | string                                                            | —            | Restrict to one gateway API key                                                          |
| `limit`       | 1–10000                                                           | `1000`       | Rows per page                                                                            |
| `offset`      | ≥ 0                                                               | `0`          | Row offset for paging                                                                    |
| `format`      | `json` \| `csv`                                                   | `json`       | `csv` returns a `text/csv` attachment with a fixed header                                |

Response (200):

```json
{
	"from": "2026-07-01",
	"to": "2026-07-31",
	"granularity": "day",
	"groupBy": ["user", "model"],
	"rows": [
		{
			"date": "2026-07-01",
			"userId": "usr_...",
			"userName": "Ada Lovelace",
			"userEmail": "ada@example.com",
			"projectId": null,
			"projectName": null,
			"apiKeyId": null,
			"apiKeyName": null,
			"model": "gpt-5.6",
			"provider": "openai",
			"requestCount": 128,
			"errorCount": 2,
			"inputTokens": 481203,
			"outputTokens": 92844,
			"totalTokens": 574047,
			"cachedTokens": 120000,
			"reasoningTokens": 18320,
			"cost": 3.4127,
			"inputCost": 2.1,
			"outputCost": 1.3127,
			"creditsRequestCount": 128,
			"apiKeysRequestCount": 0,
			"creditsCost": 3.4127,
			"apiKeysCost": 0
		}
	],
	"pagination": { "limit": 1000, "offset": 0, "hasMore": false }
}
```

Every row carries the full column set. Dimension fields you did not group by are `null`, and `date` is `null` when `granularity=total`, so the row shape stays stable for a schema-driven consumer. `creditsCost` / `apiKeysCost` split the blended `cost` into credit-billed and BYOK traffic.

### Per-member, per-model breakdown [#per-member-per-model-breakdown]

Combine the two dimensions to get the cross-tab most reporting tools want — one row per member, model, and day:

```bash
curl -G https://internal.llmgateway.io/v1/master/usage \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d from=2026-07-01 -d to=2026-07-31 \
  -d granularity=day -d groupBy=user,model \
  -d timezone=Europe/Berlin
```

### CSV export [#csv-export]

```bash
curl -G https://internal.llmgateway.io/v1/master/usage \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d from=2026-07-01 -d to=2026-07-31 \
  -d granularity=day -d groupBy=user,model -d format=csv \
  -o usage-july.csv
```

The CSV header is the same fixed column set regardless of the dimensions requested, so a scheduled export never changes shape.

<Callout type="warn">
  **Usage is attributed to the member who created the API key** (there is no
  per-caller identity on an inference request). A key shared by several people
  reports entirely under its creator. For accurate per-person reporting, issue
  one gateway API key per member — the per-member key limit under **Organization
  → Members** is the lever that enforces it.
</Callout>

Two more things worth knowing:

* These figures come from hourly rollups, not the request log, so they are **not** affected by your [data retention](https://docs.llmgateway.io/features/data-retention) setting — per-member and per-model history stays available even with retention turned off.
* Traffic from [embeddable payments](https://docs.llmgateway.io/features/embeddable-payments) end-user keys rolls up to the member who provisioned the platform key.

## IAM rules [#iam-rules]

Each gateway API key can have one or more IAM rules that restrict which models, providers, or pricing tiers it is allowed to use. Rules are evaluated at request time by the gateway. A key with no active rules has no IAM restrictions.

Rule types:

| `ruleType`        | Description                                                 |
| ----------------- | ----------------------------------------------------------- |
| `allow_models`    | Only the listed models are permitted                        |
| `deny_models`     | The listed models are blocked                               |
| `allow_providers` | Only the listed providers are permitted                     |
| `deny_providers`  | The listed providers are blocked                            |
| `allow_pricing`   | Only models matching the pricing constraint are permitted   |
| `deny_pricing`    | Models matching the pricing constraint are blocked          |
| `allow_ip_cidrs`  | Only requests from the listed IPv4/IPv6 CIDRs are permitted |
| `deny_ip_cidrs`   | Requests from the listed IPv4/IPv6 CIDRs are blocked        |

The `ruleValue` JSON object holds the rule's parameters. The fields it accepts depend on the `ruleType`:

| Field            | Type               | Used by                             |
| ---------------- | ------------------ | ----------------------------------- |
| `models`         | string\[]          | `allow_models`, `deny_models`       |
| `providers`      | string\[]          | `allow_providers`, `deny_providers` |
| `pricingType`    | `"free" \| "paid"` | `allow_pricing`, `deny_pricing`     |
| `maxInputPrice`  | number             | `allow_pricing`, `deny_pricing`     |
| `maxOutputPrice` | number             | `allow_pricing`, `deny_pricing`     |
| `ipCidrs`        | string\[]          | `allow_ip_cidrs`, `deny_ip_cidrs`   |

### IP CIDR rules [#ip-cidr-rules]

IP CIDR rules restrict gateway requests by source IP. Both IPv4 (e.g. `192.0.2.0/24`) and IPv6 (e.g. `2001:db8::/32`) ranges are supported, and you can mix both in a single rule. To restrict to a single address, use a `/32` (IPv4) or `/128` (IPv6) prefix.

The gateway reads the client IP from the first entry in the `X-Forwarded-For` header, which is set by the GCP load balancer.

IPv4-mapped IPv6 addresses (`::ffff:1.2.3.4`) are normalized to IPv4 so a single `1.2.3.0/24` rule still matches when the upstream connection happens to be IPv6.

When an `allow_ip_cidrs` rule is configured and the gateway cannot determine the client IP, the request is denied. Invalid CIDR syntax is rejected at rule-creation time with a `400` error.

All endpoints scope by the master key's organization: a `404` is returned if the API key (or rule) is not part of the authenticated master key's organization.

### List IAM rules [#list-iam-rules]

`GET /v1/master/keys/{id}/iam`

```bash
curl https://internal.llmgateway.io/v1/master/keys/ak_.../iam \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"rules": [
		{
			"id": "iam_...",
			"apiKeyId": "ak_...",
			"ruleType": "allow_models",
			"ruleValue": {
				"models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"]
			},
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Create an IAM rule [#create-an-iam-rule]

`POST /v1/master/keys/{id}/iam`

```bash
curl -X POST https://internal.llmgateway.io/v1/master/keys/ak_.../iam \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "ruleType": "allow_models",
    "ruleValue": {
      "models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"]
    }
  }'
```

Body parameters:

| Field       | Type                     | Description                                             |
| ----------- | ------------------------ | ------------------------------------------------------- |
| `ruleType`  | rule type enum (above)   | Required                                                |
| `ruleValue` | object (see table above) | Must include the fields appropriate for the chosen type |
| `status`    | `"active" \| "inactive"` | Optional, defaults to `"active"`                        |

Restricting by source IP:

```bash
curl -X POST https://internal.llmgateway.io/v1/master/keys/ak_.../iam \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "ruleType": "allow_ip_cidrs",
    "ruleValue": {
      "ipCidrs": ["192.0.2.0/24", "2001:db8::/32"]
    }
  }'
```

Response (201): the created IAM rule.

### Update an IAM rule [#update-an-iam-rule]

`PATCH /v1/master/keys/{id}/iam/{ruleId}`

All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.llmgateway.io/v1/master/keys/ak_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "inactive"
  }'
```

Body parameters (all optional, at least one required):

| Field       | Type                     | Description                             |
| ----------- | ------------------------ | --------------------------------------- |
| `ruleType`  | rule type enum (above)   | Change the rule type                    |
| `ruleValue` | object (see table above) | Replace the rule value                  |
| `status`    | `"active" \| "inactive"` | Activate or deactivate without deleting |

Response (200): the updated IAM rule.

### Delete an IAM rule [#delete-an-iam-rule]

`DELETE /v1/master/keys/{id}/iam/{ruleId}`

Permanently removes an IAM rule from the API key.

```bash
curl -X DELETE https://internal.llmgateway.io/v1/master/keys/ak_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "IAM rule deleted successfully" }
```

## Member IAM rules [#member-iam-rules]

The same rule types can be applied to an **organization member** instead of a single API key. Member-level rules are an organization-wide ceiling: a request must pass both the member's rules and the key's rules, so key rules can only narrow access further, never expand it. They apply to all regular API keys created by that member. See [Member-Level IAM Rules](https://docs.llmgateway.io/features/api-keys#member-level-iam-rules) for the full semantics.

The `{member}` path parameter accepts either the **membership id** or the member's **email address** (matched case-insensitively). Email references must resolve to a user who is a member of the master key's organization — an email belonging to a user outside the organization, or an unknown reference, returns a `404`.

The `ruleType`, `ruleValue`, and `status` fields are identical to the per-key IAM endpoints above.

### List member IAM rules [#list-member-iam-rules]

`GET /v1/master/members/{member}/iam`

```bash
curl https://internal.llmgateway.io/v1/master/members/jane@example.com/iam \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"rules": [
		{
			"id": "iam_...",
			"userOrganizationId": "uo_...",
			"ruleType": "allow_providers",
			"ruleValue": {
				"providers": ["openai"]
			},
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Create a member IAM rule [#create-a-member-iam-rule]

`POST /v1/master/members/{member}/iam`

```bash
curl -X POST https://internal.llmgateway.io/v1/master/members/jane@example.com/iam \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "ruleType": "allow_providers",
    "ruleValue": {
      "providers": ["openai"]
    }
  }'
```

Response (201): the created member IAM rule.

### Update a member IAM rule [#update-a-member-iam-rule]

`PATCH /v1/master/members/{member}/iam/{ruleId}`

All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.llmgateway.io/v1/master/members/uo_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "inactive"
  }'
```

Response (200): the updated member IAM rule.

### Delete a member IAM rule [#delete-a-member-iam-rule]

`DELETE /v1/master/members/{member}/iam/{ruleId}`

```bash
curl -X DELETE https://internal.llmgateway.io/v1/master/members/uo_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "Member IAM rule deleted successfully" }
```

## Custom providers [#custom-providers]

Custom providers are your own OpenAI-compatible endpoints, registered as BYOK provider keys with `provider: "custom"`. The gateway routes to them with the model string `custom/<name>/<model>`. See [Custom Providers](https://docs.llmgateway.io/features/custom-providers) for the dashboard flow and routing semantics.

Only custom providers are exposed through the master API — catalog providers (OpenAI, Anthropic, …) require an interactive upstream credential check and must be added from the dashboard.

All endpoints scope by the master key's organization: a `404` is returned if the provider key is not a non-deleted custom provider in that organization.

<Callout type="warning">
  The provider token is stored but never returned. Responses only include a
  `maskedToken` for identification.
</Callout>

### List custom providers [#list-custom-providers]

`GET /v1/master/custom-providers`

```bash
curl https://internal.llmgateway.io/v1/master/custom-providers \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"customProviders": [
		{
			"id": "pk_...",
			"provider": "custom",
			"name": "acme",
			"baseUrl": "https://llm.acme.example.com/v1",
			"maskedToken": "sk-a...cret",
			"status": "active",
			"customModelsOnly": true,
			"complianceAttestation": null,
			"organizationId": "org_...",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Get a custom provider [#get-a-custom-provider]

`GET /v1/master/custom-providers/{id}`

Returns a single custom provider in the master key's organization.

### Create a custom provider [#create-a-custom-provider]

`POST /v1/master/custom-providers`

```bash
curl -X POST https://internal.llmgateway.io/v1/master/custom-providers \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "acme",
    "baseUrl": "https://llm.acme.example.com/v1",
    "token": "sk-acme-...",
    "customModelsOnly": true
  }'
```

Body parameters:

| Field                   | Type               | Description                                                                                  |
| ----------------------- | ------------------ | -------------------------------------------------------------------------------------------- |
| `name`                  | string             | Lowercase letters and single hyphens only. Unique per organization; used in the model string |
| `baseUrl`               | string             | OpenAI-compatible base URL. Internal/reserved addresses are rejected                         |
| `token`                 | string             | Upstream API key. Stored server-side, never returned                                         |
| `customModelsOnly`      | boolean (optional) | Restrict the provider to models defined in your custom catalog. Default `false`              |
| `complianceAttestation` | object (optional)  | Self-attested compliance posture. `attestedAt` / `attestedByUserId` are stamped server-side  |

Response (201): the created custom provider.

### Update a custom provider [#update-a-custom-provider]

`PATCH /v1/master/custom-providers/{id}`

All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.llmgateway.io/v1/master/custom-providers/pk_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "baseUrl": "https://llm2.acme.example.com/v1",
    "token": "sk-acme-rotated-..."
  }'
```

Body parameters (all optional, at least one required):

| Field                   | Type                     | Description                                    |
| ----------------------- | ------------------------ | ---------------------------------------------- |
| `baseUrl`               | string                   | Repoint the provider at a new endpoint         |
| `token`                 | string                   | Rotate the upstream API key                    |
| `status`                | `"active" \| "inactive"` | Disable the provider without deleting it       |
| `customModelsOnly`      | boolean                  | Toggle the custom-catalog restriction          |
| `complianceAttestation` | object \| null           | Replace the attestation, or `null` to clear it |

The provider `name` is immutable — it is the routing segment your model strings reference. Create a new provider instead of renaming.

Response (200): the updated custom provider.

### Delete a custom provider [#delete-a-custom-provider]

`DELETE /v1/master/custom-providers/{id}`

Soft-deletes the provider (sets `status` to `"deleted"`). Its custom models stop resolving immediately.

Response (200):

```json
{ "message": "Custom provider deleted successfully" }
```

## Custom models [#custom-models]

Custom models are your organization's catalogue entries for a custom provider: the context window, output limit, per-token pricing, and capability flags the gateway uses to route, validate, and bill requests to that model. When a provider has `customModelsOnly: true`, only models in this catalogue can be called through it.

Query these endpoints to read the full catalogue you have registered — including `contextSize`, `maxOutput`, and every price field — for surfacing in your own dashboards or cost tooling.

<Callout type="info">
  Per-token prices are strings in USD per token. Use `e-6` notation so the
  coefficient reads directly as USD per million tokens — `"3.0e-6"` is $3.00/M.
</Callout>

### List custom models [#list-custom-models]

`GET /v1/master/custom-models`

Returns the non-deleted custom models in the master key's organization. Pass an optional `providerKeyId` query parameter to scope the list to a single custom provider.

```bash
curl "https://internal.llmgateway.io/v1/master/custom-models?providerKeyId=pk_..." \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"customModels": [
		{
			"id": "cm_...",
			"providerKeyId": "pk_...",
			"organizationId": "org_...",
			"modelName": "acme-large",
			"displayName": "Acme Large",
			"contextSize": 200000,
			"maxOutput": 32000,
			"inputPrice": "3.0e-6",
			"outputPrice": "15.0e-6",
			"cachedInputPrice": "0.3e-6",
			"streaming": "true",
			"vision": true,
			"tools": true,
			"reasoning": null,
			"jsonOutput": true,
			"supportedParameters": ["temperature", "top_p"],
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Get a custom model [#get-a-custom-model]

`GET /v1/master/custom-models/{id}`

Returns a single custom model in the master key's organization.

### Create a custom model [#create-a-custom-model]

`POST /v1/master/custom-models`

```bash
curl -X POST https://internal.llmgateway.io/v1/master/custom-models \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "providerKeyId": "pk_...",
    "modelName": "acme-large",
    "displayName": "Acme Large",
    "contextSize": 200000,
    "maxOutput": 32000,
    "inputPrice": "3.0e-6",
    "outputPrice": "15.0e-6",
    "tools": true
  }'
```

Body parameters:

| Field                    | Type                                | Description                                                |
| ------------------------ | ----------------------------------- | ---------------------------------------------------------- |
| `providerKeyId`          | string                              | Must be a custom provider in the master key's organization |
| `modelName`              | string                              | Model id sent upstream. Unique per provider                |
| `displayName`            | string (optional)                   | Human-readable label                                       |
| `contextSize`            | number (optional)                   | Context window in tokens                                   |
| `maxOutput`              | number (optional)                   | Max output tokens                                          |
| `inputPrice`             | string (optional)                   | USD per input token                                        |
| `outputPrice`            | string (optional)                   | USD per output token                                       |
| `cachedInputPrice`       | string (optional)                   | USD per cached input token                                 |
| `cacheReadInputPrice`    | string (optional)                   | USD per cache-read token                                   |
| `cacheWriteInputPrice`   | string (optional)                   | USD per cache-write token (5m TTL)                         |
| `cacheWriteInputPrice1h` | string (optional)                   | USD per cache-write token (1h TTL)                         |
| `requestPrice`           | string (optional)                   | Flat USD charged per request                               |
| `webSearchPrice`         | string (optional)                   | USD per web search                                         |
| `imageInputPrice`        | string (optional)                   | USD per image input token                                  |
| `audioInputPrice`        | string (optional)                   | USD per audio input token                                  |
| `streaming`              | `"true" \| "false" \| "only"`       | Streaming support (`"only"` = streaming-only model)        |
| `vision`                 | boolean (optional)                  | Accepts image input                                        |
| `tools`                  | boolean (optional)                  | Supports tool calling                                      |
| `reasoning`              | boolean (optional)                  | Emits reasoning output                                     |
| `jsonOutput`             | boolean (optional)                  | Supports JSON / structured output                          |
| `audio`                  | boolean (optional)                  | Accepts audio input                                        |
| `supportedParameters`    | string\[] (optional)                | Request parameters the upstream accepts                    |
| `status`                 | `"active" \| "inactive"` (optional) | Defaults to `"active"`                                     |

Response (201): the created custom model.

### Update a custom model [#update-a-custom-model]

`PATCH /v1/master/custom-models/{id}`

Accepts the same fields as create (except `providerKeyId`), all optional — provide only the ones you want to change. Renaming to a `modelName` already used by another model on the same provider returns a `400`.

```bash
curl -X PATCH https://internal.llmgateway.io/v1/master/custom-models/cm_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "contextSize": 400000, "outputPrice": "12.0e-6" }'
```

Response (200): the updated custom model.

### Delete a custom model [#delete-a-custom-model]

`DELETE /v1/master/custom-models/{id}`

Soft-deletes the model (sets `status` to `"deleted"`).

Response (200):

```json
{ "message": "Custom model deleted successfully" }
```


# Metadata
URL: https://docs.llmgateway.io/features/metadata


LLM Gateway supports sending additional metadata with your requests using custom headers. This allows you to include information like user sessions, application versions, tenant IDs, or other contextual data that can be useful for analytics and monitoring.

Later, you can filter by specific values to return, such as for a specific user or session. Additionally, in the future, you will be able to segment your analytics and monitoring based on this metadata. For example, you could show cost and latency breakdowns per user, application, country, feature, or any other dimension you want to track.

## Custom Headers [#custom-headers]

You can include custom headers with the `X-LLMGateway-` prefix to send metadata alongside your LLM requests:

```bash
curl -X POST https://api.llmgateway.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "X-LLMGateway-Country: US" \
  -H "X-LLMGateway-User-ID: 9403f741-a524-4b18-b1b2-dbb71cdff2a4" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }'
```

## Best Practices [#best-practices]

### Header Naming [#header-naming]

* Use the `X-LLMGateway-` prefix for all custom metadata
* Use descriptive, consistent naming conventions
* Avoid special characters; use hyphens to separate words

### Data Privacy [#data-privacy]

* Be mindful of sensitive data in headers
* Consider hashing or anonymizing user identifiers
* Follow your organization's data privacy policies

### Performance [#performance]

* Keep header values reasonably short
* Avoid sending unnecessary metadata that won't be used for analytics
* Consider the impact on request size, especially for high-volume applications

## Example: Multi-tenant Application [#example-multi-tenant-application]

For a multi-tenant application, you might use metadata headers like this:

```bash
curl -X POST https://api.llmgateway.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "X-LLMGateway-Tenant-ID: acme-corp" \
  -H "X-LLMGateway-User-ID: user-12345" \
  -H "X-LLMGateway-App-Version: 2.1.4" \
  -H "X-LLMGateway-Feature: chat-assistant" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Summarize this document..."
      }
    ]
  }'
```

This allows you to track usage and costs per tenant, user, application version, and feature, providing detailed insights into how your LLM integration is being used across your platform.


# Org Models Directory
URL: https://docs.llmgateway.io/features/models-directory


The **Models** page in the dashboard (`Organization → Models`) lists every model your organization can route to in a single directory: the full LLM Gateway catalogue plus the models defined for your own [custom providers](https://docs.llmgateway.io/features/custom-providers). It answers two questions for your whole team:

* **What can we call?** — every catalogue and custom model, searchable and filterable by capability, provider, price, and context size, exactly like the public [models page](https://llmgateway.io/models).
* **What are we allowed to call?** — when a [provider compliance policy](https://docs.llmgateway.io/features/compliance) is active, each provider mapping is marked eligible or blocked using the same fail-closed evaluation the gateway applies at request time.

## Who can see it [#who-can-see-it]

Every active organization member can browse the directory read-only — including project-scoped **developer** members, who see it as their only organization page. This is the recommended way for developers to discover available providers and models without being granted additional permissions. Management actions (custom-model catalog, attestations) remain restricted to organization owners and admins.

## Custom models in the directory [#custom-models-in-the-directory]

Custom models appear alongside catalogue models, addressed as `{customProvider}/{modelName}` — the exact model string to send to the gateway:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "internal-vllm/llama-4-maverick",
    "messages": [{ "role": "user", "content": "Hello!" }]
  }'
```

Their pricing, context and output limits, and capability flags come from the [custom model catalog](https://docs.llmgateway.io/features/custom-providers#custom-model-catalog). A **Source** filter switches between catalogue models, custom models, or both.

## Compliance eligibility [#compliance-eligibility]

With an enabled compliance policy (Enterprise), the directory evaluates every provider mapping against the policy — certifications, data-policy requirements, country restrictions, and provider/model allow and deny lists:

* Blocked mappings render greyed out with a ban icon; its tooltip lists the exact failing requirements (for example "No SOC 2 Type 2 report" or "Not on the allowed-providers list").
* Custom providers are evaluated against their [compliance attestation](https://docs.llmgateway.io/features/compliance#custom-providers); without one on file they are blocked while a policy is active.
* An **Eligible only** filter narrows the directory to models the gateway will actually route to.

<Callout type="info">
  The directory mirrors gateway enforcement — the same predicates decide both
  the markers on this page and the 403 responses at request time. If a model is
  marked blocked here, requests for it will be rejected while the policy is
  active.
</Callout>

## Related [#related]

* [Knowledge base → Models](https://docs.llmgateway.io/learn/models) — a walkthrough of the page itself.
* [Custom Providers](https://docs.llmgateway.io/features/custom-providers) — bring your own OpenAI-compatible endpoints and define their model catalog.
* [Compliance](https://docs.llmgateway.io/features/compliance) — configure the provider compliance policy.


# Moderations
URL: https://docs.llmgateway.io/features/moderations


LLMGateway supports the OpenAI-compatible `/v1/moderations` endpoint for text
and multimodal safety classification.

Use it when you want to:

* Screen user prompts before they reach a model
* Review generated output before displaying it
* Apply the same moderation API shape you already use with OpenAI clients

For the full request and response schema, see the
[API reference](https://docs.llmgateway.io/v1_moderations).

## Endpoint [#endpoint]

`POST https://api.llmgateway.io/v1/moderations`

Authenticate with your LLMGateway API key:

```bash
-H "Authorization: Bearer $LLM_GATEWAY_API_KEY"
```

## Supported Inputs [#supported-inputs]

The `input` field accepts:

* A single string
* An array of strings
* An array of multimodal content items with `text` and `image_url`

The default model is `omni-moderation-latest`.

## Pricing [#pricing]

Starting **Friday, August 7, 2026*&#x2A;, moderation requests are billed at a
flat **$0.00001 per request**, independent of the input size, the number of
inputs, or the moderation model you pick. Only successful requests are billed —
failed and retried attempts cost nothing. Before that date, moderation requests
are free.

Because the endpoint becomes paid, it also starts requiring a credit balance
from that date: moderation requests from an organization with no credits are
rejected with `402`. Top up before August 7 to avoid an interruption.

When your project runs in `api-keys` mode and the request is served with your
own OpenAI key, no credits are deducted and no balance is required.

## curl [#curl]

### Single text input [#single-text-input]

```bash
curl -X POST "https://api.llmgateway.io/v1/moderations" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "I want to harm someone."
  }'
```

### Multiple text inputs [#multiple-text-inputs]

```bash
curl -X POST "https://api.llmgateway.io/v1/moderations" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "omni-moderation-latest",
    "input": [
      "This is a harmless sentence.",
      "I want to attack somebody."
    ]
  }'
```

### Multimodal input [#multimodal-input]

```bash
curl -X POST "https://api.llmgateway.io/v1/moderations" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": [
      {
        "type": "text",
        "text": "Check this image for violent content."
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "https://example.com/image.png"
        }
      }
    ]
  }'
```

## OpenAI SDK [#openai-sdk]

```ts
import OpenAI from "openai";

const client = new OpenAI({
	baseURL: "https://api.llmgateway.io/v1",
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const response = await client.moderations.create({
	model: "omni-moderation-latest",
	input: "I want to harm someone.",
});

console.log(response.results[0]?.flagged);
```

## Response Shape [#response-shape]

The response follows the standard OpenAI moderation format:

```json
{
	"id": "modr-123",
	"model": "omni-moderation-latest",
	"results": [
		{
			"flagged": true,
			"categories": {
				"violence": true,
				"self_harm": false
			},
			"category_scores": {
				"violence": 0.98,
				"self_harm": 0.01
			}
		}
	]
}
```

## When To Use This Instead Of Chat Content Filtering [#when-to-use-this-instead-of-chat-content-filtering]

Use `/v1/moderations` when you want an explicit moderation decision in your own
application flow.

If you want moderation to happen automatically as part of model requests, use
LLMGateway content filtering on `/v1/chat/completions` instead.


# OCR
URL: https://docs.llmgateway.io/features/ocr


LLMGateway exposes a dedicated `/v1/ocr` endpoint for optical character
recognition. It extracts text, tables, and layout from PDFs and images and
returns them as clean markdown, one entry per page.

Use it when you want to:

* Turn scanned PDFs or photos into machine-readable markdown
* Pull structured text out of receipts, invoices, forms, or screenshots
* Feed document contents into a downstream model or RAG pipeline

For the full request and response schema, see the
[API reference](https://docs.llmgateway.io/v1_ocr).

## Endpoint [#endpoint]

`POST https://api.llmgateway.io/v1/ocr`

Authenticate with your LLMGateway API key:

```bash
-H "Authorization: Bearer $LLM_GATEWAY_API_KEY"
```

The current model is `mistral-ocr-latest&#x60;, billed at **$4 per 1,000 pages**
processed.

## Document Input [#document-input]

The `document` field accepts either a document URL (PDF) or an image:

* `{ "type": "document_url", "document_url": "https://…/file.pdf" }`
* `{ "type": "image_url", "image_url": "https://…/image.png" }`

Both `document_url` and `image_url` accept a public URL or a base64 data URL
(`data:application/pdf;base64,…` / `data:image/png;base64,…`). The `image_url`
field may also be passed as an object: `{ "url": "…" }`.

### Scoping pages [#scoping-pages]

By default the entire document is processed and every page is billed. Use the
optional `pages` field to restrict (and cap the cost of) a request:

* A list of zero-based indices: `"pages": [0, 1, 2]`
* A range string: `"pages": "0-4"`

## curl [#curl]

### Document URL [#document-url]

```bash
curl -X POST "https://api.llmgateway.io/v1/ocr" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-ocr-latest",
    "document": {
      "type": "document_url",
      "document_url": "https://arxiv.org/pdf/2201.04234"
    }
  }'
```

### Only specific pages [#only-specific-pages]

```bash
curl -X POST "https://api.llmgateway.io/v1/ocr" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-ocr-latest",
    "document": {
      "type": "document_url",
      "document_url": "https://arxiv.org/pdf/2201.04234"
    },
    "pages": "0-4"
  }'
```

### Image input [#image-input]

```bash
curl -X POST "https://api.llmgateway.io/v1/ocr" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-ocr-latest",
    "document": {
      "type": "image_url",
      "image_url": "https://example.com/receipt.png"
    }
  }'
```

### Inline (base64) document [#inline-base64-document]

```bash
BASE64_PDF=$(base64 -i invoice.pdf)

curl -X POST "https://api.llmgateway.io/v1/ocr" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"mistral-ocr-latest\",
    \"document\": {
      \"type\": \"document_url\",
      \"document_url\": \"data:application/pdf;base64,${BASE64_PDF}\"
    }
  }"
```

## Response Shape [#response-shape]

```json
{
	"pages": [
		{
			"index": 0,
			"markdown": "# Document title\n\nExtracted body text…",
			"images": [],
			"dimensions": { "dpi": 200, "height": 2200, "width": 1700 }
		}
	],
	"model": "mistral-ocr-latest",
	"document_annotation": null,
	"usage_info": {
		"pages_processed": 1,
		"doc_size_bytes": 125344
	}
}
```

Each entry in `pages` carries the markdown for one page. `usage_info.pages_processed`
reflects exactly how many pages were billed for the request.

## Billing [#billing]

OCR is billed per page processed, not per token. A request that processes 12
pages bills `12 × $0.004 = $0.048`. Scoping a request with `pages` reduces both
the work and the cost.

## OCR Models Use This Endpoint, Not Chat [#ocr-models-use-this-endpoint-not-chat]

OCR models are not chat models and cannot be called through
`/v1/chat/completions` — doing so returns a `400` pointing you here. Send OCR
requests to `/v1/ocr`.


# Organization skills
URL: https://docs.llmgateway.io/features/organization-skills


Organization skills are reusable instructions and supporting files that enterprise teams share through the CLI. Owners and admins publish skills in the dashboard; developers can discover and download them using a project API key from the same organization.

## Publish a skill [#publish-a-skill]

1. Open **Organization → Skills** in the dashboard.
2. Choose **Import SKILL.md** for a single skill, **Import folder** for a skill with references, scripts, or assets, or **New custom skill** to write instructions.
3. Review the content and save. New skills are enabled immediately.

A skill folder must contain `SKILL.md` at its root. Its YAML header needs a unique lowercase, hyphenated `name` of up to 64 characters and a `description` of up to 1,024 characters. Write the instructions below the header:

```md
---
name: code-review
description: Review code changes against the team's standards.
---

# Code review

- Check correctness and error handling.
- Explain the impact of each finding.
- Suggest tests for important edge cases.
```

Bundles support up to 100 additional files and 1 MB of total content. `SKILL.md` supports up to 200,000 characters within that limit. Supporting paths must be relative to the skill folder; duplicate paths and parent-directory traversal are rejected. Existing YAML fields, such as `license` or `allowed-tools`, are preserved.

## Manage access [#manage-access]

All active organization members can view skills. Owners and admins can edit, disable, or delete them. Keep a published skill's name unchanged; create a new skill to use a different name.

Disabling a skill removes it from CLI discovery and prevents new downloads. Deleting it also removes its stored content. These actions cannot remove copies already downloaded to a developer's machine. Publishing and management actions appear in the organization's audit log without recording the skill content.

## CLI API contract [#cli-api-contract]

These endpoints belong to the [platform API](https://internal.llmgateway.io/docs). Authenticate with `Authorization: Bearer <project-api-key>`, using a regular project API key issued to an active member with access to that project.

| Method | Path                | Response                                         |
| ------ | ------------------- | ------------------------------------------------ |
| GET    | `/v1/skills`        | `{ skills: [...] }` with enabled skill summaries |
| GET    | `/v1/skills/{name}` | `{ skill: { ...summary, content, files } }`      |

A summary contains `id`, `name`, `description`, `enabled`, `createdAt`, and `updatedAt`. Timestamps use ISO 8601. `content` is the complete `SKILL.md`, including its YAML header. Each supporting file contains a relative `path`, `content`, and optional `encoding` (`utf-8` by default, or `base64` for binary files).

Clients should discover skills at the start of a session, download selected skills by name, and write each bundle into its own directory. Revalidate authorization before reusing organization content and remove cached availability when access is denied. Treat supporting scripts as code to review before running.

The server derives the organization from the API key. Expired or revoked keys return `401`; removed project access or missing enterprise access returns `403`; missing or disabled skills return `404`. Responses use `Cache-Control: private, no-store`.

Dashboard management uses session authentication at `/orgs/{organizationId}/skills`: `GET` lists summaries and `POST` creates a bundle. At `/orgs/{organizationId}/skills/{id}`, `GET` reads a bundle, `PUT` replaces `{ content, files }`, `PATCH` sets `{ enabled }`, and `DELETE` removes it. Duplicate names return `409`.


# Realtime API
URL: https://docs.llmgateway.io/features/realtime


LLMGateway supports low-latency, speech-to-speech conversations through the
OpenAI-compatible &#x2A;*`/v1/realtime`** WebSocket endpoint. Sessions support text
and audio input/output, server-side voice activity detection (VAD), input
audio transcription, and function calling — using the same event protocol as
the OpenAI Realtime API.

The API is a drop-in replacement: point any OpenAI realtime client at
`wss://api.llmgateway.io/v1/realtime`, authenticate with your LLMGateway API
key, and keep your existing event handling.

<Callout type="info">
  Want to talk to a model right now? The [Realtime
  page](https://lounge.llmgateway.io/realtime) in Lounge runs a full voice call
  in the browser using this API.
</Callout>

## Available Models [#available-models]

Browse the available realtime models, with up-to-date pricing, on the
[models page](https://llmgateway.io/models). Realtime sessions are billed per
token — text and audio input, cached input, and output are metered separately
at the model's listed rates, matching the provider's own pricing.

Model names work like everywhere else on LLMGateway: use the plain model id
(e.g. `gpt-realtime`), a dated alias, or the `provider/model` pinned form
(e.g. `openai/gpt-realtime`).

## Connecting [#connecting]

Connect a WebSocket to:

```
wss://api.llmgateway.io/v1/realtime?model=gpt-realtime
```

Authenticate with your LLMGateway API key in the `Authorization` header (an
`x-api-key` header also works):

```javascript
import WebSocket from "ws";

const url = "wss://api.llmgateway.io/v1/realtime?model=gpt-realtime";
const ws = new WebSocket(url, {
	headers: {
		Authorization: "Bearer " + process.env.LLM_GATEWAY_API_KEY,
	},
});

ws.on("open", () => {
	console.log("Connected to server.");
});

ws.on("message", (message) => {
	const event = JSON.parse(message.toString());
	console.log(event.type);
});
```

Once connected, the session speaks the standard realtime event protocol:
send client events like `session.update`, `conversation.item.create`,
`input_audio_buffer.append`, and `response.create`; receive server events
like `session.created`, `response.output_audio.delta`, and `response.done`.

```javascript
ws.on("open", () => {
	// Configure the session.
	ws.send(
		JSON.stringify({
			type: "session.update",
			session: {
				type: "realtime",
				instructions: "You are a friendly assistant.",
				audio: {
					output: { voice: "marin" },
				},
			},
		}),
	);

	// Ask for a response.
	ws.send(
		JSON.stringify({
			type: "conversation.item.create",
			item: {
				type: "message",
				role: "user",
				content: [{ type: "input_text", text: "Say hello!" }],
			},
		}),
	);
	ws.send(JSON.stringify({ type: "response.create" }));
});
```

All events are JSON text frames; audio travels base64-encoded inside events
(binary WebSocket frames are rejected). The model is locked at connection
time — a `session.update` that tries to change `session.model` is rejected
with a `model_locked` error event.

<Callout type="warn">
  Credentials are never accepted as query parameters. A connection URL
  containing `token`, `api_key`, or `client_secret` query parameters is rejected
  with HTTP 400. Use the `Authorization` header on servers, or an ephemeral
  client secret (below) in browsers.
</Callout>

## Browser Clients and Client Secrets [#browser-clients-and-client-secrets]

Never ship a long-lived API key to a browser. Instead, mint a short-lived
**ephemeral client secret** from your backend with the OpenAI-compatible
`POST /v1/realtime/client_secrets` endpoint:

```bash
curl -X POST "https://api.llmgateway.io/v1/realtime/client_secrets" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": { "anchor": "created_at", "seconds": 120 },
    "session": {
      "type": "realtime",
      "model": "gpt-realtime"
    }
  }'
```

The response contains the secret (`ek_...`) and its expiry:

```json
{
	"value": "ek_...",
	"expires_at": 1753500000,
	"session": {
		"type": "realtime",
		"model": "gpt-realtime"
	}
}
```

The browser then connects using the standard `openai-insecure-api-key`
WebSocket subprotocol — no `Authorization` header needed:

```javascript
const ws = new WebSocket("wss://api.llmgateway.io/v1/realtime", [
	"realtime",
	"openai-insecure-api-key." + clientSecret,
]);
```

Client secret behavior:

* **TTL**: 10–300 seconds (default 60), set via `expires_after.seconds`.
  Secrets are reusable until they expire; expiry only gates opening the
  connection, not the session's duration.
* **Model pinning**: the secret is minted for one model. The `model` query
  parameter is optional when connecting with a secret; if provided, it must
  match the minted model.
* **Transcription pinning**: optionally pin an input transcription model at
  mint time via `session.audio.input.transcription.model`. The session can
  then only enable that transcription model.
* **Instruction pinning**: optionally pin the session prompt at mint time via
  `session.instructions` — see below.
* **Voice**: optionally set the session's default output voice at mint time
  via `session.audio.output.voice`. Unlike instructions this is only a
  default; the client may still change it.
* All authentication, credit, model access, and compliance checks run at
  mint time and again at connection time.

### Server-Authoritative Instructions [#server-authoritative-instructions]

By default the session prompt is set by whoever holds the socket, so a browser
client has to send `session.instructions` itself. If your server owns the
prompt, pin it at mint time instead:

```json
{
	"session": {
		"type": "realtime",
		"model": "gpt-realtime",
		"instructions": "You are a support agent for ACME."
	}
}
```

The gateway then applies the instructions itself when the session connects, so
the prompt never has to travel through the browser. It is not echoed in the
mint response, and it is stripped from the `session.created` and
`session.updated` events the client receives.

Once pinned, the instructions are locked for the session's lifetime, the same
way `session.model` is. A client that tries to change them gets an
`instructions_locked` error event and the change is not applied — this covers
both `session.update` and per-response `response.create.instructions`.

Instructions larger than 16,384 estimated tokens are rejected at mint time with
a 400 `instructions_too_large`, so an oversized prompt fails server-to-server
rather than mid-call.

<Callout type="info">
  On Gemini realtime models the same pin applies to `setup.systemInstruction`,
  and a client-supplied `systemInstruction` is rejected with the same
  `instructions_locked` code.
</Callout>

## Input Audio Transcription [#input-audio-transcription]

Enable transcription of the user's audio with `session.update`. An explicit,
supported transcription model is required — check the
[models page](https://llmgateway.io/models) for available realtime
transcription models and their pricing:

```json
{
	"type": "session.update",
	"session": {
		"type": "realtime",
		"audio": {
			"input": {
				"transcription": { "model": "gpt-4o-transcribe" }
			}
		}
	}
}
```

Transcripts arrive as standard
`conversation.item.input_audio_transcription.delta` and `.completed` events.

* The provider's implicit default (e.g. `whisper-1`) is not available;
  omitting `transcription.model` is rejected with
  `transcription_model_required`. Duration-billed transcription models cannot
  be metered per token, so only token-metered models are supported.
* The transcription model is pinned for the session on first use and cannot
  be switched afterwards; disabling transcription (`"transcription": null`)
  is always allowed.
* Transcription is billed separately from the realtime model, at the
  transcription model's listed per-token rates.
* Both the current nested form (`session.audio.input.transcription`) and the
  legacy top-level form (`session.input_audio_transcription`) are accepted.

## Function Calling [#function-calling]

Plain function tools work exactly as in the OpenAI Realtime API — declare
them in `session.update` (or per response in `response.create`), receive
`response.function_call_arguments` events, and return results with
`conversation.item.create`:

```json
{
	"type": "session.update",
	"session": {
		"type": "realtime",
		"tools": [
			{
				"type": "function",
				"name": "get_weather",
				"description": "Get the current weather for a location.",
				"parameters": {
					"type": "object",
					"properties": {
						"location": { "type": "string" }
					},
					"required": ["location"]
				}
			}
		],
		"tool_choice": "auto"
	}
}
```

Hosted tools (MCP servers, web search, code interpreter, etc.) are not yet
available and are rejected with `tool_type_not_supported`.

## Billing and Session Gating [#billing-and-session-gating]

Realtime sessions bill your organization's pay-as-you-go credits, or your own
provider key when the project uses [provider keys](https://docs.llmgateway.io/learn/provider-keys). Every
model generation — whether triggered by your `response.create` or by
server-side VAD — passes the gateway's authorization gates first (credits,
API key status, usage limits, IAM rules), so a session cannot run past an
exhausted balance. A blocked generation surfaces as a standard `error` event
on the session instead of a response.

Default per-session safety limits:

| Limit                           | Default |
| ------------------------------- | ------- |
| Maximum session duration        | 1 hour  |
| Maximum spend per session       | $10     |
| Concurrent sessions per org     | 20      |
| Concurrent sessions per API key | 10      |

Sessions that hit a limit are closed gracefully after in-flight responses are
billed. Usage appears in your [activity feed](https://llmgateway.io/dashboard)
like any other request, including separate line items for input transcription.

## Current Limitations [#current-limitations]

* **WebSocket transport only** — WebRTC and SIP are not yet supported.
* **Image input** is not yet supported in realtime sessions.
* **Hosted tools** (MCP, web search, etc.) and **stored prompt references**
  (`session.prompt`) are not supported; inline your instructions and function
  tools instead.
* Realtime requires a regular developer API key on a pay-as-you-go
  organization. End-user session tokens, platform keys, and DevPass/chat plan
  organizations are not supported yet.

## Self-Hosting [#self-hosting]

Realtime is disabled by default on self-hosted deployments. Set
`REALTIME_INLINE=true` to attach the `/v1/realtime` WebSocket listener (and
the client-secret mint endpoint) to the gateway process — `pnpm dev` sets
this automatically. Client secrets require Redis. See `.env.example` for the
tunables (session caps, concurrency limits, shutdown grace period).


# Reasoning
URL: https://docs.llmgateway.io/features/reasoning


LLMGateway supports reasoning-capable models that can show their step-by-step thought process before providing a final answer. This feature is particularly useful for complex problem-solving tasks, mathematical calculations, and logical reasoning.

## Reasoning-Enabled Models [#reasoning-enabled-models]

You can find all reasoning-enabled models on our [models page with reasoning filter](https://llmgateway.io/models?filters=1\&reasoning=true). These models include:

* OpenAI's GPT-5 series (e.g., `gpt-5`, `gpt-5-mini`)
  * Note: GPT-5 models use reasoning but currently do not return the reasoning content in the response.
* Anthropic's Claude 3.7 Sonnet
* Google's Gemini 2.0 Flash Thinking and Gemini 2.5 Pro
* GPT OSS models such as `gpt-oss-120b` and `gpt-oss-20b`
* Z.AI's reasoning models

<Callout type="info">
  Some models may reason internally even if the `reasoning_effort` parameter is
  not specified.
</Callout>

## Using the Reasoning Parameter [#using-the-reasoning-parameter]

There are two ways to control reasoning effort:

### Option 1: Top-level `reasoning_effort` [#option-1-top-level-reasoning_effort]

Add the `reasoning_effort` parameter directly to your request:

* `none` - Disable reasoning. Supported by OpenAI's newer reasoning models (e.g. `gpt-5.4-mini` and later, which accept `none` instead of `minimal`). For other providers this turns reasoning off.
* `minimal` - Fastest reasoning with minimal thought process (only for GPT-5 models)
* `low` - Light reasoning for simpler tasks
* `medium` - Balanced reasoning for most tasks
* `high` - Deep reasoning for complex problems
* `xhigh` - Very deep reasoning for the most complex problems
* `max` - Highest reasoning tier, above `xhigh`. Supported by Anthropic thinking models and OpenAI GPT-5.6 models. Effort tiers are never downgraded by the gateway: providers that accept an effort parameter receive the value unchanged (unsupported values result in a provider error), while providers that take a thinking budget instead (Anthropic, Google, Alibaba) have each tier translated to a native budget

<Callout type="info">
  OpenAI's reasoning models do not all accept the same effort values. The
  original GPT-5 models support `minimal`, while newer models (e.g.
  `gpt-5.4-mini` and later) replace it with `none`. If you send an effort value
  the target model doesn't support, OpenAI returns an `unsupported_value` error.
  The exact values each provider mapping accepts are exposed as
  `reasoning_efforts` on the [`/v1/models`](https://api.llmgateway.io/v1/models)
  endpoint.
</Callout>

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {
        "role": "user",
        "content": "What is 2/3 + 1/4 + 5/6?"
      }
    ],
    "reasoning_effort": "medium"
  }'
```

### Option 2: Using the `reasoning` object [#option-2-using-the-reasoning-object]

Use the unified `reasoning` configuration object with an `effort` field:

* `none` - Disable reasoning
* `minimal` - Fastest reasoning with minimal thought process
* `low` - Light reasoning for simpler tasks
* `medium` - Balanced reasoning for most tasks
* `high` - Deep reasoning for complex problems
* `xhigh` - Very deep reasoning for the most complex problems
* `max` - Highest reasoning tier, above `xhigh`. Supported by Anthropic thinking models and OpenAI GPT-5.6 models. Effort tiers are never downgraded by the gateway: providers that accept an effort parameter receive the value unchanged (unsupported values result in a provider error), while providers that take a thinking budget instead (Anthropic, Google, Alibaba) have each tier translated to a native budget

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {
        "role": "user",
        "content": "What is 2/3 + 1/4 + 5/6?"
      }
    ],
    "reasoning": {
      "effort": "medium"
    }
  }'
```

<Callout type="warning">
  You cannot use both `reasoning_effort` and `reasoning.effort` in the same
  request. Choose one approach. However, you can combine `reasoning_effort` or
  `reasoning.effort` with `reasoning.max_tokens` — when `max_tokens` is
  specified, it takes priority over the effort level.
</Callout>

### Example Response [#example-response]

The response will include a `reasoning` field in the message object containing the model's step-by-step thought process:

```json
{
	"id": "chatcmpl-abc123",
	"object": "chat.completion",
	"created": 1234567890,
	"model": "gpt-oss-120b",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "The answer is 1.75 or 7/4.",
				"reasoning": "First, I need to find a common denominator for 2/3, 1/4, and 5/6. The LCD is 12. Converting: 2/3 = 8/12, 1/4 = 3/12, 5/6 = 10/12. Adding: 8/12 + 3/12 + 10/12 = 21/12 = 1.75 or 7/4."
			},
			"finish_reason": "completed"
		}
	],
	"usage": {
		"prompt_tokens": 20,
		"completion_tokens": 45,
		"reasoning_tokens": 35,
		"total_tokens": 65
	}
}
```

## Specifying Reasoning Token Budget [#specifying-reasoning-token-budget]

For models that support it, you can specify an exact token budget for reasoning using the `reasoning` object with `max_tokens`. This gives you precise control over how many tokens the model allocates to its thinking process.

<Callout type="info">
  When `reasoning.max_tokens` is specified, it overrides `reasoning.effort` and
  `reasoning_effort`. Supported by Anthropic Claude and Google Gemini thinking
  models, plus Alibaba-hosted thinking models (forwarded as DashScope's
  `thinking_budget`).
</Callout>

### Example Request [#example-request]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-4-20250514",
    "messages": [
      {
        "role": "user",
        "content": "Explain the P vs NP problem and why it matters."
      }
    ],
    "reasoning": {
      "max_tokens": 8000
    }
  }'
```

### Supported Models [#supported-models]

The `reasoning.max_tokens` parameter is supported by:

* **Anthropic Claude**: Claude 3.7 Sonnet, Claude Sonnet 4, Claude Opus 4, Claude Opus 4.5
* **Google Gemini**: Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 3 Pro Preview

When using auto-routing or canonical models with `reasoning.max_tokens`, only providers that support this feature will be considered.

### Provider-Specific Constraints [#provider-specific-constraints]

* **Anthropic**: Reasoning budget must be between 1,024 and 128,000 tokens. Values outside this range are automatically clamped.
* **Google**: No specific constraints on the reasoning budget.

### Error Handling [#error-handling]

If you specify `reasoning.max_tokens` for a model that doesn't support it, you'll receive an error:

```json
{
	"error": {
		"message": "Model gpt-4o does not support reasoning.max_tokens. Remove the reasoning parameter or use a model that supports explicit reasoning token budgets.",
		"type": "invalid_request_error",
		"code": "model_not_supported"
	}
}
```

## Controlling Response Verbosity [#controlling-response-verbosity]

For OpenAI GPT-5 and later models, you can control how detailed the model's final answer is with the top-level `verbosity` parameter. This is independent of `reasoning_effort`: `reasoning_effort` controls how much the model thinks, while `verbosity` controls how much it writes in its response.

Accepted values:

* `low` - Concise responses with minimal elaboration
* `medium` - Balanced level of detail
* `high` - Detailed, thorough responses

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {
        "role": "user",
        "content": "Explain how a hash map works."
      }
    ],
    "verbosity": "low"
  }'
```

<Callout type="info">
  `verbosity` is only supported by OpenAI GPT-5 and later models. You can check
  which model mappings accept it via the
  [`/v1/models`](https://api.llmgateway.io/v1/models) endpoint. It can be
  combined freely with `reasoning_effort` or the `reasoning` object.
</Callout>

### Error Handling [#error-handling-1]

If you specify `verbosity` for a model that doesn't support it, you'll receive a `400` error:

```json
{
	"error": {
		"message": "Model gpt-4o does not support the verbosity parameter. Remove the verbosity parameter or use a model that supports it (OpenAI GPT-5 and later).",
		"type": "invalid_request_error",
		"code": "model_not_supported"
	}
}
```

## Streaming Reasoning Content [#streaming-reasoning-content]

When streaming is enabled, reasoning content will be streamed as part of the response chunks:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {
        "role": "user",
        "content": "Solve this logic puzzle: If all roses are flowers and some flowers fade quickly, can we conclude that some roses fade quickly?"
      }
    ],
    "reasoning_effort": "high",
    "stream": true
  }'
```

The reasoning content will appear in the stream chunks before the final answer, allowing you to display the model's thought process in real-time.

Example:

```
data: {
	"id": "chatcmpl-fb266880-1016-4797-9a70-f21a538edaf6",
	"object": "chat.completion.chunk",
	"created": 1761048126,
	"model": "openai/gpt-oss-20b",
	"choices": [
		{
			"index": 0,
			"delta": {
				"reasoning": "It's ",
				"role": "assistant"
			},
			"finish_reason": null
		}
	]
}
```

## Preserving Reasoning Across Calls [#preserving-reasoning-across-calls]

Agents work over many turns: the model reasons, calls a tool, and the tool result arrives in the next request. Reasoning models are trained to continue from the reasoning they produced on earlier turns, so dropping it mid-task degrades tool-calling and multi-turn performance.

LLMGateway preserves reasoning the same way OpenAI's Responses API does. The model returns an opaque **encrypted reasoning** payload, and you send it back unchanged along with the rest of the conversation on the next call. The payload is issued and verified by the provider — the gateway passes it through without inspecting or rewriting it.

<Callout type="info">
  Encrypted reasoning is produced by models served over OpenAI's Responses API
  (OpenAI and Azure OpenAI reasoning models). Other reasoning models return
  summary text only, and there is nothing to replay — asking for encrypted
  reasoning is ignored rather than rejected, so the same client code works
  against every model. Gemini thought signatures are handled automatically: the
  gateway re-attaches them to replayed tool calls.
</Callout>

### Responses API [#responses-api]

Ask for the payloads with `include: ["reasoning.encrypted_content"]`, and send `store: false` if you want the turn to stay stateless:

```bash
curl -X POST "https://api.llmgateway.io/v1/responses" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4-nano",
    "input": [{ "role": "user", "content": "What is the weather in Paris? Use the tool." }],
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "parameters": {
          "type": "object",
          "properties": { "city": { "type": "string" } },
          "required": ["city"]
        }
      }
    ],
    "store": false,
    "include": ["reasoning.encrypted_content"],
    "reasoning": { "effort": "high" }
  }'
```

Reasoning items in `output` then carry `encrypted_content`:

```json
{
	"type": "reasoning",
	"id": "rs_057d93d1ba22357e01...",
	"summary": [],
	"encrypted_content": "gAAAAABqfkEO3uEFtXeGSNE9..."
}
```

On the next turn, replay the previous `output` items unchanged — reasoning items included — followed by your tool results:

```json
{
	"model": "gpt-5.4-nano",
	"input": [
		{
			"role": "user",
			"content": "What is the weather in Paris? Use the tool."
		},
		{
			"type": "reasoning",
			"id": "rs_057d93d1ba22357e01...",
			"summary": [],
			"encrypted_content": "gAAAAABqfkEO3uEFtXeGSNE9..."
		},
		{
			"type": "function_call",
			"call_id": "call_923JIrDbCvIHnGqfvDbPeaLh",
			"name": "get_weather",
			"arguments": "{\"city\":\"Paris\"}"
		},
		{
			"type": "function_call_output",
			"call_id": "call_923JIrDbCvIHnGqfvDbPeaLh",
			"output": "18C sunny"
		}
	],
	"store": false,
	"include": ["reasoning.encrypted_content"],
	"reasoning": { "effort": "high" }
}
```

Without the `include` value, reasoning items come back without `encrypted_content` and there is nothing to replay. When streaming, the payload arrives on the reasoning item's `response.output_item.done` event.

#### Stateful Alternative [#stateful-alternative]

If you would rather not carry the payloads yourself, leave `store` at its default (`true`) and chain turns with `previous_response_id`. The gateway keeps the stored response — encrypted reasoning included — for [30 days](https://docs.llmgateway.io/features/data-retention#retention-periods) and replays the reasoning for you, so `include` is not needed on that path.

### Chat Completions [#chat-completions]

The Chat Completions format has no reasoning items, so the gateway carries the same payloads on the assistant message as `reasoning_details`:

```json
{
	"role": "assistant",
	"content": null,
	"tool_calls": [
		{
			"id": "call_923JIrDbCvIHnGqfvDbPeaLh",
			"type": "function",
			"function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
		}
	],
	"reasoning_details": [
		{
			"type": "reasoning.encrypted",
			"data": "gAAAAABqfkEO3uEFtXeGSNE9...",
			"id": "rs_057d93d1ba22357e01...",
			"format": "openai-responses-v1",
			"index": 0
		}
	]
}
```

To preserve the reasoning, append the assistant message back into `messages` exactly as you received it (with `reasoning_details` intact) before the matching `tool` result. No opt-in parameter is required — the field is always present when the model produced one. When streaming, the entries arrive as a `delta.reasoning_details` chunk.

Replaying is safe across models: entries a provider cannot consume are stripped before the upstream request, so the same conversation history can be sent to any model.

### Error Handling [#error-handling-2]

Payloads are verified by the provider that issued them. A modified, truncated, or foreign payload is rejected upstream and the error is forwarded unchanged:

```json
{
	"error": {
		"message": "The encrypted content for item rs_057d93d1ba22357e01... could not be verified. Reason: Encrypted content could not be decrypted or parsed.",
		"type": "invalid_request_error",
		"code": "invalid_encrypted_content"
	}
}
```

Only replay payloads against the model and provider that produced them.

## Usage Tracking [#usage-tracking]

### Response Payload [#response-payload]

The `usage` object in the response includes reasoning-specific token counts:

* `reasoning_tokens` - Number of tokens used for the reasoning process
* `completion_tokens` - Number of tokens in the final answer
* `prompt_tokens` - Number of tokens in the input
* `total_tokens` - Sum of all token counts

### Logs and Analytics [#logs-and-analytics]

All requests using the `reasoning_effort` parameter are tracked in your dashboard logs with:

* The `reasoningContent` field containing the full reasoning text
* Separate token counts for reasoning vs. completion
* Performance metrics for reasoning-enabled requests

You can view detailed logs for each request in the [dashboard](https://llmgateway.io/dashboard) to analyze how models are reasoning through problems.

## Auto-Routing with Reasoning [#auto-routing-with-reasoning]

When using auto-routing (specifying a model like `gpt-5` without a specific version), LLMGateway will:

1. Automatically set `reasoning_effort` to `minimal` for GPT-5 models
2. Set `reasoning_effort` to `low` for other auto-routed reasoning models
3. Only route to providers that support reasoning when `reasoning_effort` is specified

This ensures optimal performance and cost when using auto-routing with reasoning-capable models.

## Model-Specific Behavior [#model-specific-behavior]

Not all reasoning models return reasoning content in the same way. Some models (like OpenAI models) may reason internally but not expose the reasoning content in the response. LLMGateway makes sure the response is unified across different providers, but the depth and format of reasoning may vary.

## Best Practices [#best-practices]

1. **Choose appropriate reasoning effort**: Use `low` or `minimal` for simple tasks, `medium` for most tasks, and `high` only for complex problems that require deep reasoning
2. **Monitor token usage**: Reasoning can significantly increase token consumption - monitor your `reasoning_tokens` in the usage object
3. **Stream for better UX**: When building user-facing applications, enable streaming to show the reasoning process in real-time
4. **Check logs**: Review the `reasoningContent` in your dashboard logs to understand how models are solving problems

## Error Handling [#error-handling-3]

If you specify `reasoning_effort` for a model that doesn't support reasoning, you'll receive an error:

```json
{
	"error": {
		"message": "Model gpt-4o does not support reasoning. Remove the reasoning_effort parameter or use a reasoning-capable model.",
		"type": "invalid_request_error",
		"code": "model_not_supported"
	}
}
```

To avoid this error, only use the `reasoning_effort` parameter with [reasoning-enabled models](https://llmgateway.io/models?filters=1\&reasoning=true).


# Rerank
URL: https://docs.llmgateway.io/features/rerank


LLMGateway exposes a Cohere-compatible `/v1/rerank` endpoint that scores a list
of candidate documents against a query and returns them ordered by relevance.

Rerankers are cross-encoders: they read the query and a document *together*
rather than embedding each in isolation. That makes them far more accurate than
vector similarity, but too slow to run over a whole corpus. The usual pattern is
two-stage retrieval — use [embeddings](https://docs.llmgateway.io/features/embeddings) to cheaply fetch
the top \~100 candidates, then rerank those down to the handful you actually put
in the prompt.

Browse available rerank models on the
[models page](https://llmgateway.io/models?filters=1\&rerank=true).

For the full request and response schema, see the
[API reference](https://docs.llmgateway.io/v1_rerank).

## Endpoint [#endpoint]

`POST https://api.llmgateway.io/v1/rerank`

## cURL [#curl]

```bash
curl -X POST "https://api.llmgateway.io/v1/rerank" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-reranker-8b",
    "query": "What is the capital of France?",
    "documents": [
      "Paris is the capital of France.",
      "Berlin is the capital of Germany.",
      "Madrid is the capital of Spain."
    ],
    "top_n": 2
  }'
```

```json
{
	"id": "kK1sT9pQ2mR7vX4nB6cL8dF3gH5jW0yZaA2eS4uI",
	"results": [
		{ "index": 0, "relevance_score": 0.9781517386436462 },
		{ "index": 1, "relevance_score": 0.00010548105638008565 }
	],
	"meta": {
		"api_version": { "version": "1" },
		"billed_units": { "input_tokens": 255, "total_tokens": 255 }
	}
}
```

Results are sorted by `relevance_score` descending. Each `index` refers back to
the position of that document in the request's `documents` array, so you can map
scores onto your own records without relying on the returned text.

The response `id` echoes the `x-request-id` header when you send one, which is
handy for correlating a rerank call with its entry in the activity log.

## Request fields [#request-fields]

| Field                | Type       | Description                                                               |
| -------------------- | ---------- | ------------------------------------------------------------------------- |
| `model`              | `string`   | Rerank model to use. Optionally prefixed with a provider (`deepinfra/…`). |
| `query`              | `string`   | The search query to rank documents against.                               |
| `documents`          | `string[]` | Candidate documents to score. At least one.                               |
| `top_n`              | `number`   | Return only the top N results. Defaults to all documents.                 |
| `return_documents`   | `boolean`  | Include the document text on each result. Defaults to `false`.            |
| `max_chunks_per_doc` | `number`   | Maximum chunks per document. Accepted, but not every provider honors it.  |

## Two-stage retrieval [#two-stage-retrieval]

```ts
// 1. Cheap recall: embed the query and pull the top 100 candidates
//    from your vector store.
const candidates = await vectorStore.search(queryEmbedding, { limit: 100 });

// 2. Precise ordering: rerank those candidates and keep the best 5.
const res = await fetch("https://api.llmgateway.io/v1/rerank", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.LLM_GATEWAY_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "qwen3-reranker-8b",
		query,
		documents: candidates.map((c) => c.text),
		top_n: 5,
	}),
});

const { results } = await res.json();
const topDocuments = results.map((r) => candidates[r.index]);
```

<Callout type="info">
  Rerank models are billed on input tokens only — the query and every document
  you send are scored as input. There are no output tokens, since the response
  is a list of scores rather than generated text. Sending 100 long documents
  costs real money, so keep the first-stage candidate set tight.
</Callout>

<Callout type="warn">
  Rerank models only work on `/v1/rerank`. Requesting one on
  `/v1/chat/completions` returns a 400 pointing you at the right endpoint, and
  they are not available in the playground.
</Callout>


# Response Healing
URL: https://docs.llmgateway.io/features/response-healing


Response Healing is a plugin that automatically validates and repairs malformed JSON responses from AI models. When enabled, LLM Gateway ensures that API responses conform to your specified schemas even when the model's formatting is imperfect.

## Why Response Healing? [#why-response-healing]

Large language models occasionally produce invalid JSON, especially in complex scenarios:

* **Markdown wrapping**: Models often wrap JSON in code blocks like \`\`\`json...\`\`\`
* **Mixed content**: JSON may be preceded or followed by explanatory text
* **Syntax errors**: Trailing commas, unquoted keys, or single quotes instead of double quotes
* **Truncated output**: Token limits may cut off responses mid-JSON

Response Healing automatically detects and fixes these issues, saving you from implementing error handling for every possible malformed response.

## Enabling Response Healing [#enabling-response-healing]

To enable Response Healing, add `response-healing` to the `plugins` array in your request:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Return a JSON object with name and age"}],
    "response_format": {"type": "json_object"},
    "plugins": [{"id": "response-healing"}]
  }'
```

<Callout type="info">
  Response Healing only activates when `response_format` is set to `json_object`
  or `json_schema`. For regular text responses, the plugin has no effect.
</Callout>

## How It Works [#how-it-works]

When Response Healing is enabled, LLM Gateway applies a series of repair strategies to malformed JSON responses:

### 1. Markdown Extraction [#1-markdown-extraction]

Extracts JSON from markdown code blocks:

```text
Here's the data:
\`\`\`json
{"name": "Alice", "age": 30}
\`\`\`
```

Becomes:

```json
{ "name": "Alice", "age": 30 }
```

### 2. Mixed Content Extraction [#2-mixed-content-extraction]

Separates JSON from surrounding text:

```text
Sure! Here is the JSON you requested: {"name": "Alice", "age": 30} Let me know if you need anything else.
```

Becomes:

```json
{ "name": "Alice", "age": 30 }
```

### 3. Syntax Fixes [#3-syntax-fixes]

Repairs common JSON syntax violations:

| Issue           | Before              | After               |
| --------------- | ------------------- | ------------------- |
| Trailing commas | `{"a": 1,}`         | `{"a": 1}`          |
| Unquoted keys   | `{name: "Alice"}`   | `{"name": "Alice"}` |
| Single quotes   | `{'name': 'Alice'}` | `{"name": "Alice"}` |

### 4. Truncation Completion [#4-truncation-completion]

Adds missing closing brackets for truncated responses:

```text
{"name": "Alice", "data": {"nested": true
```

Becomes:

```json
{ "name": "Alice", "data": { "nested": true } }
```

## Usage Examples [#usage-examples]

### With JSON Object Format [#with-json-object-format]

Request a structured response with automatic healing:

```typescript
const response = await fetch("https://api.llmgateway.io/v1/chat/completions", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.LLM_GATEWAY_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "gpt-4o",
		messages: [
			{
				role: "user",
				content:
					"Return a JSON object with fields: name (string) and age (number)",
			},
		],
		response_format: { type: "json_object" },
		plugins: [{ id: "response-healing" }],
	}),
});

const result = await response.json();
// Response is guaranteed to be valid JSON
const data = JSON.parse(result.choices[0].message.content);
```

### With JSON Schema [#with-json-schema]

For stricter validation, combine with `json_schema`:

```typescript
const response = await fetch("https://api.llmgateway.io/v1/chat/completions", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.LLM_GATEWAY_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "gpt-4o",
		messages: [
			{
				role: "user",
				content: "Generate a user profile",
			},
		],
		response_format: {
			type: "json_schema",
			json_schema: {
				name: "user_profile",
				schema: {
					type: "object",
					required: ["name", "email"],
					properties: {
						name: { type: "string" },
						email: { type: "string" },
						age: { type: "number" },
					},
				},
			},
		},
		plugins: [{ id: "response-healing" }],
	}),
});

const result = await response.json();
```

## Healing Metadata [#healing-metadata]

When a response is healed, the healing method is logged for debugging. The following healing methods may be applied:

| Method                     | Description                                 |
| -------------------------- | ------------------------------------------- |
| `markdown_extraction`      | JSON extracted from markdown code blocks    |
| `mixed_content_extraction` | JSON extracted from surrounding text        |
| `syntax_fix`               | Trailing commas, quotes, or keys were fixed |
| `truncation_completion`    | Missing closing brackets were added         |
| `combined_strategies`      | Multiple strategies were applied            |

## Limitations [#limitations]

<Callout type="warning">
  Response Healing is only available for non-streaming requests. Streaming
  responses are returned as-is without healing.
</Callout>

Response Healing works best for:

* Simple to moderately complex JSON structures
* Common formatting issues from LLMs

It may not be able to repair:

* Severely corrupted or nonsensical output
* Complex nested structures with multiple issues
* Responses that don't contain any recognizable JSON

## Best Practices [#best-practices]

### Use with Structured Prompts [#use-with-structured-prompts]

Combine Response Healing with clear instructions for best results:

```typescript
const response = await fetch("https://api.llmgateway.io/v1/chat/completions", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.LLM_GATEWAY_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "gpt-4o",
		messages: [
			{
				role: "system",
				content: "Always respond with valid JSON. No explanations.",
			},
			{
				role: "user",
				content: "List three colors as a JSON array",
			},
		],
		response_format: { type: "json_object" },
		plugins: [{ id: "response-healing" }],
	}),
});

const result = await response.json();
```

### Validate Critical Data [#validate-critical-data]

For critical applications, validate the healed JSON in your code:

```typescript
const result = await response.json();
const content = result.choices[0].message.content;
const data = JSON.parse(content);

// Add your own validation
if (!data.name || typeof data.name !== "string") {
	throw new Error("Invalid response: missing name");
}
```

### Monitor Healing Rates [#monitor-healing-rates]

If you notice frequent healing in your logs, consider:

* Improving your prompts to request cleaner JSON
* Using models with better JSON output (e.g., GPT-4o, Claude 3.5)
* Adding explicit JSON examples in your prompts


# Routing
URL: https://docs.llmgateway.io/features/routing


LLMGateway provides flexible and intelligent routing options to help you get the best performance and cost efficiency from your AI applications. Whether you want to use specific models, providers, or let our system automatically optimize your requests, we've got you covered.

LLMGateway also includes **automatic retry and fallback** — if a provider fails, your request is seamlessly retried on the next best provider, all within the same API call.

## Model Selection [#model-selection]

### Any Model Name [#any-model-name]

You can use any model name from our [models page](https://llmgateway.io/models) or discover available models programmatically through the [/v1/models endpoint](https://docs.llmgateway.io/v1_models).

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

### Model ID Routing [#model-id-routing]

Choose a specific model ID to route to the **best available provider** for that model. LLMGateway's smart routing algorithm considers multiple factors to find the optimal provider across all configured options.

#### Smart Routing Algorithm [#smart-routing-algorithm]

When you use a model ID without a provider prefix, LLMGateway's intelligent routing system analyzes multiple factors to select the best provider.

**Weighted Scoring System**:

Each factor has a **relative weight**. The factors are scored as ratios against the best provider in the candidate set (e.g. a provider that is twice as expensive as the cheapest scores `1.0` on price), and each ratio is multiplied by its weight divided by the sum of all active weights. The provider with the lowest (best) total score wins.

The default weights are:

| Factor          | Default weight | Notes                                                                      |
| --------------- | -------------- | -------------------------------------------------------------------------- |
| **Price**       | `0.6`          | Cost efficiency (average of input and output price)                        |
| **Uptime**      | `0.5`          | Provider reliability / low error rate                                      |
| **Throughput**  | `0.05`         | Tokens per second generation speed                                         |
| **Latency**     | `0.025`        | Time to first token — **only applied for streaming requests**              |
| **Cache**       | `0.2`          | Prompt-cache support — **only applied for large prompts** (≥ 5,000 tokens) |
| **Image price** | `1.0`          | Replaces the price weight for image-generation models                      |

Because the weights are relative and normalized by the sum of the active weights, price and uptime dominate routing decisions in practice, while throughput and latency act as tie-breakers between otherwise comparable providers.

**Latency Weight for Non-Streaming Requests**:

The latency weight only applies to streaming requests (time-to-first-token is only measured there). For non-streaming requests the latency weight is dropped and its share is redistributed proportionally across the remaining factors.

**Time-Decayed Metrics Window**:

Provider metrics (uptime, throughput, latency) are not a flat "last N minutes" snapshot. They are aggregated over a rolling **60-minute window** with a time-decay weighting so very recent behavior dominates while older data still contributes:

* The most recent **1 minute** is weighted &#x2A;*10×**
* The most recent **5 minutes** are weighted &#x2A;*3×**
* The remainder of the 60-minute window is weighted &#x2A;*1×**

This makes routing react quickly to a provider that just started failing or slowing down, without overreacting to a single noisy data point.

**Cache Support for Large Prompts**:

When the estimated prompt is at least 5,000 tokens, the **cache weight** (default `0.2`) is factored into the score based on whether each provider supports prompt caching (advertised via a cached input price). Providers that support caching score better than ones that do not, since caching can substantially reduce the cost of large or repeated prompts. Below the 5,000-token threshold, this weight is dropped entirely — caching has little impact on small prompts, so cache support is ignored. The selected provider's cache support is exposed as `cacheSupported` on the routing metadata.

For these cache-relevant requests, the price each provider is ranked on also reflects what a cached workload actually pays: the input price is blended with the provider's cached input price at an assumed cache-hit rate (`thresholds.cacheHitRate`, default `0.7`), and output is weighted at an assumed output:input ratio (`thresholds.cacheOutputRatio`, default `0.2`) instead of parity, since large-prompt traffic is dominated by input tokens. Providers that do not publish a cached input price are ranked at their full input price, matching how they bill. Setting `cacheHitRate` to `0` and `cacheOutputRatio` to `1` restores pure list-price ranking.

**Exponential Uptime Penalty**:

Providers with uptime below 95% receive an additional exponential penalty that increases rapidly as uptime drops:

* 95-100% uptime: No penalty
* 90% uptime: \~0.07 penalty
* 80% uptime: \~0.62 penalty
* 70% uptime: \~1.73 penalty
* 50% uptime: \~5.61 penalty

This ensures providers experiencing significant issues are strongly deprioritized while minor fluctuations have minimal impact. The penalty threshold (default `95%`) is configurable.

**Provider Priority**:

Each provider has a **priority** value (default `1`) that nudges routing toward or away from it independently of live metrics:

* A provider's priority is applied as a `(1 - priority)` adjustment to its score — higher priority lowers the score (more preferred), lower priority raises it (less preferred).
* A priority of **0** disables the provider entirely, removing it from routing for that model.

Provider priorities are surfaced in the routing metadata so you can see how they influenced a decision.

**Epsilon-Greedy Exploration** (1% of requests by default):

To solve the "cold start problem" where new or unused providers never get traffic to build up metrics, the system randomly explores different providers a small fraction of the time (default 1%, configurable). This ensures:

* All providers periodically receive traffic
* New providers can prove their reliability
* The system adapts to changing provider performance
* You benefit from improved routing decisions over time

The exploration rate is configurable per project through the routing configuration (`thresholds.explorationRate`), and self-hosted deployments can override it globally with the `EXPLORATION_RATE` environment variable (a number between `0` and `1`).

**Stable Provider Preference**:

To avoid unnecessary churn between providers that score similarly, LLMGateway remembers the best provider chosen for each model and sticks with it across requests — even if another provider edges ahead slightly on the next score calculation.

On every routing decision, the system checks whether the previously selected provider is still acceptable:

* **Uptime hard switch**: if the preferred provider's uptime drops below &#x2A;*85%**, routing switches to the current best-scoring provider immediately.
* **Score margin soft switch**: the preferred provider is replaced only when a better option's score is more than **0.15** ahead. Small fluctuations caused by metric noise or minor price differences do not trigger a switch.
* **Periodic re-evaluation**: the preference expires after **1 hour**, at which point the next request picks the best-scoring provider fresh and stores it as the new preferred.

Requests that are part of the epsilon-greedy exploration bypass this preference entirely so that all providers continue to receive periodic traffic and build up metrics.

The selection reason in routing metadata will show `stable-preferred` when a request was served by the stored preference rather than the top-scored provider at that moment.

<Callout type="info">
  Self-hosted deployments can tune this behavior with three environment
  variables: `PREFERRED_PROVIDER_TTL` (preference lifetime in seconds, default
  `3600`), `PREFERRED_PROVIDER_UPTIME_THRESHOLD` (hard-switch uptime floor,
  default `85`), and `PREFERRED_PROVIDER_SCORE_MARGIN` (soft-switch score gap,
  default `0.15`). On the **Enterprise plan**, these same values can be
  customized per project from the dashboard — see [Per-Project Routing
  Configuration](#per-project-routing-configuration-enterprise).
</Callout>

**Routing Metadata**:

Every request includes detailed routing metadata in the logs, showing:

* Available providers that were considered
* Selected provider and selection reason
* Scores for each provider (including uptime, throughput, latency, price, priority, and cache support)

This transparency allows you to understand and debug routing decisions.

<Callout type="info">
  Using model IDs without a provider prefix automatically routes to the optimal
  provider based on reliability, speed, and cost. The system continuously learns
  and adapts based on real-time performance metrics.
</Callout>

<Callout type="success">
  Smart routing prioritizes reliability over cost, ensuring your requests are
  routed to providers with proven uptime and performance, while still
  considering cost efficiency.
</Callout>

### Routing Strategy [#routing-strategy]

By default, model-ID routing uses the full weighted score described above (`routing: "auto"`). When you care about a single dimension, set the `routing` field — named after the factor it optimizes — to bias provider selection toward it:

| Strategy                     | Behavior                                                                             |
| ---------------------------- | ------------------------------------------------------------------------------------ |
| `auto&#x60; &#x2A;(default)* | Full weighted smart-routing score (price, uptime, throughput, latency, cache).       |
| `price`                      | Gives price a **90% relative weight**, so the cheapest provider almost always wins.  |
| `throughput`                 | Gives throughput a **90% relative weight**, so the fastest-generating provider wins. |
| `latency`                    | Gives latency a **90% relative weight**, so the lowest time-to-first-token wins.     |

Each non-`auto` strategy keeps a small (10%) uptime weight, and the [exponential uptime penalty](#smart-routing-algorithm) still applies on top. This means the dominant pick is still skipped in favor of another provider when it has extremely bad uptime — you get the cheapest (or fastest) provider that is actually healthy, not one that is effectively down.

Because time-to-first-token is only measured for streaming requests, `routing: "latency"` only biases streaming requests; for non-streaming requests it falls back to selecting on uptime.

```bash
# Always pick the cheapest healthy provider for this model
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}],
    "routing": "price"
  }'
```

```bash
# Always pick the highest-throughput healthy provider for this model
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}],
    "routing": "throughput"
  }'
```

<Callout type="info">
  The `routing` field only applies to model-id routing. Combining it with a
  specific provider (e.g. `openai/gpt-4o`) returns a `400` error, since the
  strategy can't influence a pinned provider — remove the provider prefix to use
  a strategy. On **coding (dev) plans**, only `auto` and `price` are allowed;
  the other strategies return a `400` error because they would bypass the
  prompt-cache–aware routing those plans depend on.
</Callout>

### Sticky Session Routing [#sticky-session-routing]

When a model is served by multiple providers, every request is normally scored independently — so a multi-turn conversation can bounce between providers. That defeats provider-side **prompt caching**, which only pays off when consecutive requests with a shared prefix hit the **same** provider.

Sticky session routing solves this: attach a session identifier and LLMGateway pins all requests for that session to a single provider (and region), keeping the upstream prompt cache warm across the whole conversation.

#### Setting the session id [#setting-the-session-id]

For chat completions, the session key is resolved in priority order:

1. The `x-session-id` header
2. The `prompt_cache_key` body field (OpenAI-compatible)
3. The `user` body field (OpenAI-compatible)

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-session-id: conversation-9f8e7d6c" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

For the Anthropic Messages endpoint (`/v1/messages`), the session key is derived automatically from `metadata.user_id` — coding agents such as Claude Code embed the session id there — and forwarded internally. An explicit `x-session-id` header still takes precedence.

#### How pinning works [#how-pinning-works]

On a session's **first** request the provider is chosen by the normal weighted smart-routing score — the same price-, priority-, uptime-, and throughput-aware algorithm used for non-sticky requests. That choice is then **persisted for the session** and reused on every subsequent request, so the upstream prompt cache stays warm without bouncing the conversation between providers.

Because the pinned provider is replayed directly, sticky requests **skip the epsilon-greedy exploration** — a session is never randomly bounced to a different provider mid-conversation.

Request compatibility takes precedence over the saved pin. The gateway first filters mappings for requirements such as input modalities, service tiers, regions, and a non-`auto` `tool_choice`, then looks for the pinned provider in that eligible set. If the pinned mapping cannot honor the request but another mapping can, the session moves to the capable mapping and the pin is updated. For a fixed model or dynamic route where **no** mapping can honor `tool_choice`, the gateway preserves availability instead: it keeps the mappings, downgrades `tool_choice` to `auto`, and sticky routing may retain the existing pin. Automatic model selection does not use that fallback because it can choose a capable model instead.

#### Falling back when a provider is down [#falling-back-when-a-provider-is-down]

An established pin yields only when its provider can no longer serve the session well. A session is re-scored and re-pinned to the current weighted-best provider when its provider:

* Drops below the session uptime threshold (default 85%),
* Is filtered out by health checks (e.g. excluded for low uptime), or
* Fails the request and is dropped by the [automatic retry & fallback](#automatic-retry--fallback) loop.

Re-pinning runs the same weighted algorithm again, so the replacement is the best currently available provider — not an arbitrary one.

The selection reason in routing metadata shows `session-sticky` when a request was pinned via a session id.

<Callout type="info">
  Sticky routing optimizes for cache locality over per-request churn. Once a
  session is pinned it stays on its provider even if a cheaper or faster
  alternative becomes momentarily available, since the prompt-cache savings
  typically outweigh the difference — but the initial pick still respects price
  and priority. Requests without a session id are unaffected and continue to use
  the weighted smart-routing algorithm.
</Callout>

### Provider-Specific Routing [#provider-specific-routing]

To use a specific provider without any fallbacks, prefix the model name with the provider name followed by a slash:

```bash
# Use OpenAI specifically
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Use DeepSeek provider specifically
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek/deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

<Callout type="info">
  Provider-specific routing is not available on **DevPass coding plans** —
  provider-prefixed model ids (and custom provider routing) return a `403`
  there. DevPass always uses the prompt-cache–aware smart routing above with
  plain model ids. Provider pinning requires the pay-as-you-go API. See
  [Provider routing on
  DevPass](https://docs.llmgateway.io/learn/model-categories#provider-routing-on-devpass).
</Callout>

#### Regions [#regions]

Some providers expose the same model in multiple regions. In that case, LLMGateway supports two routing modes:

* `provider/model` selects the best eligible region for that provider using the same routing inputs used elsewhere: recent uptime, throughput, latency, and price
* `provider/model:region` pins the request to one exact region

```bash
# Let LLMGateway choose the best Alibaba region for DeepSeek V3.2
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "alibaba/deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Force a specific Alibaba region
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "alibaba/deepseek-v3.2:cn-beijing",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

If your provider key stores an explicit region, that region acts like a lock and LLMGateway will only use that region for provider-specific requests. If no explicit region is configured on the provider key, provider-specific requests can still score all eligible regions for that provider.

Routing metadata reflects this:

* Dynamic provider-region selection shows all eligible regional scores that were considered
* Explicitly pinned regions show only the pinned region in the score list

<Callout type="info">
  Region-aware routing only compares regions that are actually available for the
  current project mode and provider setup. In credits mode, that means only
  regions backed by configured environment keys. In API keys and hybrid mode, an
  explicit provider-key region restricts the request to that region.
</Callout>

A few regions are served by an endpoint that belongs to your own account rather
than a shared one — Alibaba Cloud's EU (Frankfurt) region has no shared
DashScope domain and is served by your Model Studio workspace's dedicated host.
Such a region still works from an API key alone, via the provider's shared entry
point, but that endpoint is rate-limited and carries no SLA. Set the workspace
ID on the provider key (copy it from the API Host shown when you create the key)
to route through your own endpoint instead.

#### Low-Uptime Protection [#low-uptime-protection]

When you specify a provider explicitly, LLMGateway checks the provider's recent uptime (from the time-decayed metrics window described above). If the uptime falls below 90%, the system automatically routes your request to the best available alternative provider to ensure reliability. This protects your application from providers experiencing temporary issues. The fallback threshold (default `90%`) is configurable.

<Callout type="info">
  If the requested provider has low uptime but no alternative providers are
  available for that model, the request will still be sent to the originally
  requested provider.
</Callout>

#### Disabling Fallback with X-No-Fallback Header [#disabling-fallback-with-x-no-fallback-header]

If you need to bypass this protection and always use the exact provider you specified regardless of its current uptime, you can use the `X-No-Fallback` header:

```bash
# Force use of a specific provider even if it has low uptime
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-No-Fallback: true" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

<Callout type="warn">
  Using `X-No-Fallback: true` disables automatic provider failover. Your
  requests will be sent to the specified provider even if it is experiencing
  issues, which may result in higher error rates. Retries may still occur
  against another key for the same provider when multiple keys are configured.
</Callout>

When the `X-No-Fallback` header is used, the routing metadata in logs will include `noFallback: true` to indicate that fallback was disabled for that request.

## Automatic Retry & Fallback [#automatic-retry--fallback]

When using model ID routing (without a provider prefix), LLMGateway automatically retries failed requests on alternate providers. This happens transparently within the same API call — your application receives the successful response as if nothing went wrong.

### How Retry Works [#how-retry-works]

1. Your request is routed to the best available provider using the smart routing algorithm
2. If that provider returns a server error (5xx), times out, or has a connection failure, the gateway marks the provider as failed
3. The next best available provider is selected and the request is retried
4. Up to **2 retries** are attempted before returning an error to the client

```
Request → Provider A (500 error) → Provider B (200 OK) → Response
```

Both streaming and non-streaming requests support automatic retry.

### What Triggers a Retry [#what-triggers-a-retry]

Retries are triggered by **server-side failures** only:

* **5xx errors** (500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, etc.)
* **Timeouts** (upstream provider took too long to respond)
* **Connection failures** (network errors, DNS failures, etc.)

Retries are **not** triggered by:

* **4xx client errors** (400 Bad Request, 401 Unauthorized, 403 Forbidden, 422 Unprocessable Entity)
* **Content filter responses** (Azure ResponsibleAI, etc.)

### When Retry Is Disabled [#when-retry-is-disabled]

Automatic retry to a different provider is disabled when:

* The `X-No-Fallback: true` header is set
* A specific provider is requested (e.g., `openai/gpt-4o`)
* No alternative providers are available for the requested model
* The maximum retry count (2) has been exhausted

Retries can still happen within the same provider when multiple keys are
configured and the current key fails with a retryable error.

### Routing Transparency [#routing-transparency]

Every provider attempt — both failed and successful — is recorded in the `routing` array in the response metadata (streaming and non-streaming alike) and activity logs:

```json
{
	"metadata": {
		"routing": [
			{
				"provider": "openai",
				"model": "gpt-4o",
				"status_code": 500,
				"error_type": "server_error",
				"succeeded": false,
				"credentialSource": "byok",
				"apiKeyHash": "f029ee9",
				"providerKeyId": "pk_2f9a...",
				"providerKeyLabel": "billing-team-key"
			},
			{
				"provider": "azure",
				"model": "gpt-4o",
				"status_code": 200,
				"error_type": "none",
				"succeeded": true,
				"credentialSource": "platform",
				"apiKeyHash": "ecb88d5"
			}
		]
	}
}
```

#### Whose key served each attempt [#whose-key-served-each-attempt]

`credentialSource` says who owns the provider credential an attempt was sent with:

| Value      | Meaning                                                                                                   |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| `byok`     | Your own provider key. The provider bills you directly and the attempt is not deducted from your credits. |
| `platform` | An LLM Gateway credential. The attempt runs on credits and is deducted from your balance.                 |

This matters most in **hybrid** mode, where a request that fails on your own key falls back to LLM Gateway's credential: both attempts appear in the same `routing` array, and only `credentialSource` tells them apart — `apiKeyHash` is an opaque fingerprint that says two attempts used different keys, not which key was yours. The same value is stored on the log as `routingMetadata.usedCredentialSource` for the credential that ultimately served the request, and is shown as a **your key** / **LLM Gateway key** badge in the dashboard's routing view.

#### Which of your keys ran [#which-of-your-keys-ran]

A `byok` attempt also carries the key itself: `providerKeyId`, and `providerKeyLabel` — the key as it is named on your [provider keys](https://llmgateway.io/dashboard) page (its name, or its masked token when it has none). So when several of your keys are configured for a provider and the gateway rotates between them, each attempt says which one it used instead of leaving you to decode a fingerprint.

Chat requests additionally record `routingMetadata.eligibleProviderKeys` on the log: your keys that were candidates for the provider that served the request, in selection order. It is omitted for credits-mode projects, which route on LLM Gateway credentials, and for custom providers, whose keys are scoped by their own catalogue.

<Callout type="info">
  These fields describe **your** keys only. LLM Gateway's own credentials — the
  ones that serve credits-mode traffic — are never named: a `platform` attempt
  still reports `credentialSource` and `apiKeyHash`, but never `providerKeyId`
  or `providerKeyLabel`.
</Callout>

### Retried Log Tracking [#retried-log-tracking]

Each provider attempt creates its own log entry. Failed attempts that were retried are marked with:

* **`retried: true`** — indicates this failed request was retried on another provider
* **`retriedByLogId`** — the ID of the final successful log entry

This allows you to distinguish between unrecovered failures and failures that were transparently recovered via retry. In the dashboard, retried logs display a "Retried" badge with a link to the successful log.

### Impact on Provider Health [#impact-on-provider-health]

Failed attempts still count against the provider's uptime score, even when the request was successfully retried on another provider. This means:

* A provider that keeps failing will see its uptime score drop
* The exponential uptime penalty kicks in below 95% (see [Smart Routing Algorithm](#smart-routing-algorithm))
* Future requests are automatically routed away from unreliable providers
* Your application stays reliable without any code changes on your side

<Callout type="success">
  Automatic retry and fallback works together with smart routing to provide
  self-healing behavior. Failing providers are automatically avoided, and your
  requests are transparently recovered on reliable alternatives.
</Callout>

## Per-Project Routing Configuration (Enterprise) [#per-project-routing-configuration-enterprise]

The values described above — scoring weights, thresholds, retry behavior, the metrics window, sticky-routing, and per-provider priorities — are the **defaults** that apply to every project. On the **Enterprise plan**, you can override any of them **per project** from the dashboard under **Project Settings → Routing**. Projects on other plans always use the defaults.

Overrides are merged on top of the defaults, so you only set the values you want to change. When a custom configuration is disabled, the project falls back to the defaults.

The following groups can be customized per project:

| Group                   | What it controls                                                                                                                                                                     | Defaults                                                                                                                                                                             |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Weights**             | Relative importance of each scoring factor                                                                                                                                           | `price 0.6`, `imagePrice 1.0`, `uptime 0.5`, `throughput 0.05`, `latency 0.025`, `cache 0.2`                                                                                         |
| **Thresholds**          | Cache prompt-size threshold, uptime-penalty threshold, exploration rate, and the assumed defaults used when no metrics exist                                                         | `cachePromptTokens 5000`, `cacheHitRate 0.7`, `cacheOutputRatio 0.2`, `uptimePenalty 95`, `defaultUptime 100`, `defaultLatency 1000`, `defaultThroughput 50`, `explorationRate 0.01` |
| **Retry**               | Max cross-provider fallback attempts and the low-uptime reroute threshold                                                                                                            | `maxRetries 2`, `lowUptimeFallbackThreshold 90`                                                                                                                                      |
| **Timeouts**            | Per-request time limits (end-to-end, streaming, non-streaming) — see [Request Timeouts](https://docs.llmgateway.io/features/timeouts). Capped at the infrastructure defaults — an override can only lower them | `gatewayMs 1,500,000`, `streamingMs 1,200,000`, `plainMs 600,000`                                                                                                                    |
| **History**             | The metrics window and the time-decay tier boundaries and weights                                                                                                                    | `windowMinutes 60` (max 120), `tier1Minutes 1`, `tier2Minutes 5`, `tier1Weight 10`, `tier2Weight 3`, `tier3Weight 1`                                                                 |
| **Sticky**              | Stable-provider preference: on/off, TTL, hard-switch uptime floor, soft-switch score margin                                                                                          | `enabled true`, `ttlSeconds 3600`, `uptimeThreshold 85`, `scoreMargin 0.15`                                                                                                          |
| **Provider priorities** | Per-provider priority multipliers; set a provider to `0` to disable it for that project                                                                                              | `1` for every provider                                                                                                                                                               |

<Callout type="info">
  Per-project routing configuration requires the Enterprise plan. If you'd like
  to tune routing for your workloads, contact us at [contact@llmgateway.io](mailto:contact@llmgateway.io).
</Callout>

## Optimized Auto Routing [#optimized-auto-routing]

Auto routing automatically selects the best model for your specific use case without you having to specify a model at all.

### Current Implementation [#current-implementation]

The auto routing system currently:

* **Chooses cost-effective models** by default for optimal price-to-performance ratio
* **Automatically scales to more powerful models** based on your request's context size
* **Handles large contexts intelligently** by selecting models with appropriate context windows

```bash
# Let LLMGateway choose the optimal model
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Your request here..."}]
  }'
```

### Free Models Only [#free-models-only]

When using auto routing, you can restrict the selection to only free models (models with zero input and output pricing) by setting the `free_models_only` parameter to `true`:

```bash
# Auto route to free models only
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "free_models_only": true
  }'
```

<Callout type="success">
  Adding even a small amount of credits to your account (e.g., $10) will
  immediately upgrade your free model rate limits from 5 requests per 10 minutes
  to 20 requests per minute.
</Callout>

<Callout type="info">
  The `free_models_only` parameter only works with auto routing (`"model":
  	"auto"`). If no free models are available that meet your request requirements,
  the API will return an error.
</Callout>

### Reasoning models only [#reasoning-models-only]

Just specify the `reasoning_effort` value and only a model which supports reasoning will be chosen. This parameter is not specific to the auto model.

```bash
# Auto route only to reasoning models
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "reasoning_effort": "medium"
  }'
```

### Exclude Reasoning Models [#exclude-reasoning-models]

When using auto routing, you can exclude reasoning models from selection by setting the `no_reasoning` parameter to `true`. This is useful when you want faster responses or need to avoid the additional cost and latency of reasoning models:

```bash
# Auto route excluding reasoning models
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "no_reasoning": true
  }'
```

<Callout type="info">
  The `no_reasoning` parameter only works with auto routing (`"model": "auto"`).
  If no non-reasoning models are available that meet your request requirements,
  the API will return an error.
</Callout>

<Callout type="success">
  Auto routing analyzes your payload and automatically chooses between
  cost-effective models for simple requests and more powerful models for complex
  or large-context requests.
</Callout>

### Coming Soon: Advanced Optimization [#coming-soon-advanced-optimization]

We're continuously improving our auto routing capabilities. Soon you'll benefit from:

* **Tool call optimization**: Automatically select models that excel at function calling and structured outputs
* **Content-aware routing**: Analyze message content to determine the best model for specific types of requests (coding, creative writing, analysis, etc.)
* **Performance-based routing**: Route based on historical performance data for similar requests
* **Multi-model orchestration**: Intelligently combine multiple models for complex workflows

### How It Works [#how-it-works]

1. **Request Analysis**: The system analyzes your request including message content, context size, and any special parameters
2. **Model Selection**: Based on the analysis, it selects the most appropriate model considering cost, performance, and capabilities
3. **Transparent Routing**: Your request is seamlessly routed to the chosen model and provider
4. **Optimized Response**: You receive the best possible response while maintaining cost efficiency

<Callout type="info">
  Auto routing decisions are transparent in your usage logs, so you can always
  see which model was selected for each request.
</Callout>

## Best Practices [#best-practices]

### For Development [#for-development]

* Use specific model names during development and testing
* Leverage auto routing for production workloads to optimize costs

### For Production [#for-production]

* Use auto routing (`"model": "auto"`) for the best balance of cost and performance
* Monitor your usage patterns through the dashboard to understand routing decisions
* Set up provider keys for multiple providers to maximize routing options

### For Cost Optimization [#for-cost-optimization]

* Let auto routing handle model selection to automatically use the most cost-effective options
* Use model IDs without provider prefixes to always get the cheapest available provider
* Monitor your usage analytics to track cost savings from intelligent routing


# Service Tiers
URL: https://docs.llmgateway.io/features/service-tiers


Some OpenAI, Google, and Fireworks models support selectable **processing tiers** that trade
latency and availability against price. You pick one per request with the
OpenAI-compatible `service_tier` parameter, and LLM Gateway forwards it only
when the selected provider/model mapping supports that tier.

| Tier         | `service_tier`            | Cost vs. standard | Latency / availability                      |
| ------------ | ------------------------- | ----------------- | ------------------------------------------- |
| Standard     | `default` / `auto` / omit | baseline          | Normal on-demand latency                    |
| **Flex**     | `flex`                    | **−50%**          | Best-effort; may be preempted under load    |
| **Priority** | `priority`                | varies by model   | Prioritized above standard and flex traffic |

## Using the `service_tier` parameter [#using-the-service_tier-parameter]

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google-vertex/gemini-3.1-pro-preview",
    "service_tier": "priority",
    "messages": [
      { "role": "user", "content": "Summarize this incident report." }
    ]
  }'
```

Accepted values are `flex`, `priority`, and `default`/`auto` (standard). If you
request `flex` or `priority` for a provider/model mapping that does not support
that tier, the gateway returns a 400 `unsupported_service_tier` error and logs
the request as a client error.

The parameter works the same on the OpenAI-compatible **Responses API**
(`/v1/responses`): the tier is forwarded to the provider and the response's
`service_tier` field echoes the tier that was actually served.

Coding (dev) plans are limited to `default`/`auto` and `flex` — the premium
`priority` tier is not part of the plan and a request that asks for it returns a 403.

## Supported providers [#supported-providers]

Service tiers are explicit per provider/model mapping. Check the model page for
the exact tiers exposed by each provider card.

* **OpenAI** (`openai`) — sent as the OpenAI `service_tier` request field for
  supported OpenAI models. Flex is billed at 0.5x standard token prices and
  Priority uses the model-specific multiplier shown on the model page.

* **Google Vertex AI** (`google-vertex`) — sent as the
  `X-Vertex-AI-LLM-Shared-Request-Type` request header, together with
  `X-Vertex-AI-LLM-Request-Type: shared` so the request bypasses any Provisioned
  Throughput on the project and actually reaches the shared Flex/Priority tier.
  Flex and Priority are served only on the **global** endpoint, which is the
  gateway default. Google Flex PayGo applies a 0.5x multiplier; Google Priority
  PayGo applies a 1.8x multiplier.

* **Google AI Studio / Gemini API** (`google-ai-studio`) — sent as a
  `service_tier` field in the request body for configured models that opt in.

* **Fireworks AI** (`fireworks`) — sent as a `service_tier` field in the request
  body on its OpenAI-compatible chat completions endpoint. Only Priority is
  offered (Fireworks publishes no Flex rate card), billed at a 1.25x multiplier.
  Fireworks does not report the tier it served, but a Priority request is either
  served at Priority or shed with a 503, so an accepted request is billed at the
  tier it was sent at.

<Callout type="info">
  Tiers are supported on a **subset** of models, and the Flex and Priority
  subsets differ by provider. For example, Google Flex PayGo lists Gemini 3
  image / Nano Banana models, but Google Priority PayGo does not; those
  configured image mappings are Flex-only.
</Callout>

<Callout type="warn">
  Flex and Priority are only honored when the request reaches the provider
  directly, so a provider key with a **custom base URL** (a proxy) is excluded
  from service-tier routing — a proxy may silently drop the tier and serve
  standard. This applies to every provider that offers tiers, including OpenAI:
  the tier travels as a `service_tier` body field that an OpenAI-compatible
  proxy is free to ignore. With multiple providers/keys, the gateway routes
  around the ineligible key automatically; if a request pins a provider whose
  only key uses a custom base URL, it returns a 400 instead of silently
  downgrading. Keys with no custom base URL (the managed default) are always
  eligible.
</Callout>

## Retries and fallback never downgrade the tier [#retries-and-fallback-never-downgrade-the-tier]

A request can change provider or credential mid-flight: LLM Gateway falls back
to another provider when one fails, and rotates to another key for the same
provider when a credential returns a 429 or an auth error. A requested tier is
carried through all of it.

* Provider routing is narrowed to mappings that support the requested tier
  **before** a provider is picked, so no fallback candidate can be one that
  would serve the request as standard.
* Key selection — BYOK keys, platform-managed credentials, and env credentials
  alike — skips any credential that cannot carry the tier (a proxy base URL, or
  a Vertex credential pinned to a regional endpoint), on the first attempt and
  on every retry.
* Every attempt re-resolves the tier against the provider, region and
  credential it actually resolved to, and fails rather than sending at a lower
  tier. If no eligible candidate is left, you get the upstream error instead of
  a silently downgraded response.

This applies to a tier you requested yourself. The optional coding-plan default
tier is a cost preference rather than a requirement, so a request that cannot be
served at that tier runs at standard instead of failing; `used_service_tier` in
the response metadata always reports what was actually served. A tier sent on
the request always takes precedence over that default, within the tiers the plan
allows.

<Callout type="info">
  Rotating to another key for the same provider is a separate upstream account
  as far as prompt caching is concerned, so a retried request re-writes its
  cached prefix rather than reading the original one. Send `x-no-fallback: true`
  to have the original upstream error returned to you — note that this disables
  cross-provider fallback, not key rotation within a provider.
</Callout>

## Pricing uses multipliers [#pricing-uses-multipliers]

Service tiers do not define separate model prices in LLM Gateway. They multiply
the provider mapping's standard token prices:

* Standard / `default` / `auto`: 1x
* Flex: 0.5x
* Priority: model/provider-specific, shown on the model page

The multiplier scales per-token costs, including input, output, cached, and
image tokens. Flat per-request and web-search fees are not tier-scaled.

## Billing follows the served tier [#billing-follows-the-served-tier]

When a provider reports the tier that was actually served, LLM Gateway bills
that returned tier instead of blindly billing the requested value:

* A `priority` request that runs as priority is billed at 2.5x.
* A `flex` request that runs as flex is billed at 0.5x.
* A request that is served as standard is billed at the standard 1x rate.

The served tier is read back from the provider response — Vertex reports it in
`usageMetadata.trafficType` (`ON_DEMAND_PRIORITY` / `ON_DEMAND_FLEX` /
`ON_DEMAND`), Google AI Studio reports it in the `x-gemini-service-tier`
response header, and OpenAI can return `service_tier` in response payloads or
stream events. Providers that report no tier at all (Fireworks) never downgrade
silently — they reject the request instead — so an accepted request is billed at
the tier it was sent at.

<Callout type="warn">
  LLM Gateway rejects unsupported tier requests before provider routing. For
  example, `gemini-3-pro-image` exposes Flex and Priority for Google AI Studio,
  but only Flex for Vertex. Other mappings expose neither tier.
</Callout>

You can see per-tier pricing for each model on its
[model page](https://llmgateway.io/models). Supported provider cards include a
Service Tier selector in the card header and show the active multiplier next to
each tier.

## Sources [#sources]

* [OpenAI API pricing](https://openai.com/api/pricing/)
* [Google Flex PayGo](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/flex-paygo)
* [Google Priority PayGo](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/priority-paygo)
* [Fireworks Serverless Priority and Fast](https://docs.fireworks.ai/serverless/priority-and-fast)


# Sessions
URL: https://docs.llmgateway.io/features/sessions


A **session** ties together the requests that belong to the same conversation or workflow. By attaching a stable session identifier to your requests, LLMGateway can treat them as a unit — keeping provider routing consistent across turns and letting you trace and filter the whole conversation in the dashboard.

Sessions are the foundation for several features. Today they power **sticky provider routing** and **session-level observability**; more session-scoped capabilities will build on the same identifier over time.

## Setting the session id [#setting-the-session-id]

For chat completions, the session key is resolved in priority order — the first present value wins:

1. The `x-session-id` header
2. The `x-session-affinity` header (sent automatically by coding agents such as opencode)
3. The `prompt_cache_key` body field (OpenAI-compatible)
4. The `user` body field (OpenAI-compatible)

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-session-id: conversation-9f8e7d6c" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

Reuse the same session id for every request in a conversation. If you don't set any of the values above, the request simply has no session and behaves exactly as before.

### Anthropic Messages endpoint [#anthropic-messages-endpoint]

For the [Anthropic Messages endpoint](https://docs.llmgateway.io/features/anthropic-endpoint) (`/v1/messages`), the session key is derived automatically from `metadata.user_id`. Coding agents such as Claude Code send a JSON object there (e.g. `{"session_id":"<uuid>",…}`); the gateway uses its `session_id` field. An explicit `x-session-id` header still takes precedence.

## Sticky provider routing [#sticky-provider-routing]

When a model is served by multiple providers, requests are normally scored independently, so a multi-turn conversation can bounce between providers. That defeats provider-side **prompt caching**, which only pays off when consecutive requests with a shared prefix reach the **same** provider.

With a session id set, LLMGateway scores the session's first request with the normal weighted smart-routing algorithm (price, priority, uptime, throughput) and then **pins that provider for the session**, reusing it on every subsequent request to keep the prompt cache warm. The session stays on that provider — skipping the epsilon-greedy exploration — and only moves when its provider drops below the session uptime threshold or leaves the available pool (health filtering or a failed request dropped by retry/fallback), at which point the session is re-scored and re-pinned to the current best provider.

See [Routing → Sticky Session Routing](https://docs.llmgateway.io/features/routing) for the full algorithm, fallback behavior, and the `session-sticky` routing-metadata reason.

Session stickiness is **on by default**. Enterprise projects can turn it off per project under **Settings → Routing → Session Stickiness**; when disabled, every request is scored independently regardless of session id (the id is still recorded for observability).

<Callout type="info">
  Sticky routing optimizes for cache locality over per-request price. A session
  stays on its provider even if a cheaper or faster alternative is momentarily
  available, since the prompt-cache savings typically outweigh the difference.
</Callout>

## Upstream prompt-cache routing [#upstream-prompt-cache-routing]

Some providers use an OpenAI-style `prompt_cache_key` to route requests to the cache shard that already holds your prompt prefix — without it, repeat requests can land on different backends and miss the cache entirely (Meta requires it for cache hits in practice; OpenAI and Azure use it to improve hit rates under load).

When a request has a session id and you didn't send a `prompt_cache_key` yourself, LLMGateway forwards a **keyed hash** (HMAC-SHA256 with a gateway-side secret) of the session id as the `prompt_cache_key` to providers that support it (currently OpenAI, Azure, and Meta). Hashing means your raw session ids are never exposed to providers; the hash is stable per session, which is all cache routing needs. A `prompt_cache_key` you set explicitly takes precedence and is forwarded as-is on those same surfaces.

On provider surfaces that don't support the field, no key is sent at all — whether derived or explicit. This currently applies to Sakana (the field is not part of its API) and to Azure chat-completions requests, which can be served by legacy deployment-based API versions that reject unknown body fields; Azure requests on the Responses API always carry the key. Providers not listed above use different caching mechanisms (for example Anthropic `cache_control` breakpoints or Google implicit caching), so the `prompt_cache_key` doesn't apply to them either.

For Meta, requests without any session id still get a cache key derived from the conversation's first messages, so multi-turn conversations hit Meta's prompt cache even when no session signal is present.

## Observing sessions in the activity log [#observing-sessions-in-the-activity-log]

Every request is logged with its resolved session id. In the dashboard **Activity** view you can:

* See the **Session ID** on each request's metadata, alongside the request and trace IDs.
* **Filter by session id** using the search field next to the custom-metadata search, to pull up every request that belongs to a conversation in one place.

This makes it easy to follow a full conversation end-to-end — inspecting how each turn was routed, what it cost, and which provider served it.

<Callout type="info">
  The session id is distinct from freeform [metadata](https://docs.llmgateway.io/features/metadata). Use
  metadata custom headers for arbitrary tags (user, tenant, app version); use
  the session id for the one value that should keep a conversation pinned and
  traceable.
</Callout>


# Source Attribution
URL: https://docs.llmgateway.io/features/source


The `X-Source` header allows you to identify your domain when making requests to LLM Gateway. This information is used to generate public usage statistics showing how LLM Gateway is being used across different websites and applications.

## X-Source Header [#x-source-header]

Include the `X-Source` header with your domain name in your requests:

```bash
curl -X POST https://api.llmgateway.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "X-Source: example.com" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }'
```

## Domain Format [#domain-format]

The `X-Source` header accepts domain names in various formats. All of the following are valid and will be normalized to the same domain:

* `example.com`
* `https://example.com`
* `https://www.example.com`
* `www.example.com`

All variations will be stripped down to the base domain (`example.com`) for aggregation purposes.

## Public Statistics [#public-statistics]

Data from the `X-Source` header is used to generate public statistics about LLM Gateway usage, including:

* **Popular Domains**: Which websites and applications are using LLM Gateway most frequently
* **Model Usage**: What models are being used by different domains
* **Geographic Distribution**: Where requests are coming from across different sources
* **Growth Trends**: How usage is growing over time for different domains

These statistics help demonstrate the adoption and impact of LLM Gateway across the ecosystem.

## Privacy Considerations [#privacy-considerations]

### What's Public [#whats-public]

* Domain names (stripped of protocol and www prefixes)
* Aggregated request counts and model usage
* General geographic regions (country-level data)

### What's Private [#whats-private]

* Individual request content or responses
* User identifiers or personal information
* Detailed usage patterns beyond aggregated counts
* API keys or authentication details

## Benefits [#benefits]

Including the `X-Source` header provides several benefits:

### For Your Project [#for-your-project]

* **Recognition**: Your domain will appear in public usage statistics
* **Credibility**: Demonstrates real-world usage of your application
* **Community**: Contributes to the broader LLM Gateway ecosystem

### For the Community [#for-the-community]

* **Transparency**: Shows real adoption and usage patterns
* **Inspiration**: Other developers can see successful implementations
* **Growth**: Helps demonstrate the value of open-source LLM infrastructure

## Optional but Recommended [#optional-but-recommended]

While the `X-Source` header is optional, we strongly encourage its use to:

* Support transparency in the LLM Gateway ecosystem
* Help showcase successful integrations
* Contribute to understanding of LLM usage patterns
* Demonstrate the real-world impact of your application

Your participation helps build a more transparent and collaborative LLM ecosystem.


# Speech Generation
URL: https://docs.llmgateway.io/features/speech-generation


LLMGateway supports text-to-speech (TTS) through the OpenAI-compatible
&#x2A;*`/v1/audio/speech`** endpoint, powered by ElevenLabs, Google Gemini, OpenAI,
and Alibaba Qwen speech models.

<Callout type="info">
  Want to hear the voices before writing code? The [Audio
  Studio](https://lounge.llmgateway.io/audio) in Lounge generates speech from up
  to three models side by side, with per-model voice, format, and speed
  controls.
</Callout>

## Available Models [#available-models]

Browse all speech generation models, with up-to-date pricing, on the
[models page](https://llmgateway.io/models?filters=1\&audioGeneration=true).

<Callout type="info">
  Billing varies by model family. Some models are billed on token usage reported
  by the provider (input text tokens and output audio tokens), while others are
  billed on input character count (those return audio bytes without usage data).
  See the [models
  page](https://llmgateway.io/models?filters=1\&audioGeneration=true) for each
  model's exact pricing.
</Callout>

## Parameters [#parameters]

| Parameter         | Type   | Default  | Description                                                                                                                                                                     |
| ----------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`           | string | required | The speech model to use                                                                                                                                                         |
| `input`           | string | required | The text to synthesize into speech                                                                                                                                              |
| `voice`           | string | model    | A prebuilt voice. Defaults to `Kore` (Gemini), `alloy` (OpenAI), `Sarah` (ElevenLabs), or the model's first voice on Qwen (`longanlingxin` on Plus, `longanhuan_v3.6` on Flash) |
| `response_format` | string | model    | Audio format. OpenAI: `mp3` (default), `opus`, `aac`, `flac`, `wav`, `pcm`. ElevenLabs: `mp3` (default), `wav`, `pcm`, `opus`. Gemini: `wav` (default), `pcm`. Qwen: `wav`      |
| `instructions`    | string | —        | Optional style/delivery directive prepended to the input (e.g. `"Say cheerfully"`)                                                                                              |
| `speed`           | number | —        | Accepted for OpenAI compatibility, but not applied by Gemini speech models                                                                                                      |

<Callout type="info">
  Gemini speech models return raw PCM audio. LLMGateway wraps it in a WAV
  container by default (`response_format: "wav"`), or returns the raw 16-bit
  little-endian PCM at 24 kHz when `response_format: "pcm"` is requested.
  Other formats such as `mp3` are only available on the OpenAI models, which
  return the audio already encoded in the requested format.
</Callout>

## curl [#curl]

```bash
curl -X POST "https://api.llmgateway.io/v1/audio/speech" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-2.5-flash-preview-tts",
    "input": "Hello, welcome to LLM Gateway!",
    "voice": "Kore"
  }' \
  --output speech.wav
```

## OpenAI SDK [#openai-sdk]

Works with the standard OpenAI client library — just point the base URL to
LLMGateway.

```ts
import OpenAI from "openai";
import { writeFileSync } from "fs";

const openai = new OpenAI({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
	baseURL: "https://api.llmgateway.io/v1",
});

const response = await openai.audio.speech.create({
	model: "gemini-2.5-flash-preview-tts",
	voice: "Kore",
	input: "Hello, welcome to LLM Gateway!",
});

const buffer = Buffer.from(await response.arrayBuffer());
writeFileSync("speech.wav", buffer);
```

## Streaming [#streaming]

<Callout type="info">
  Streaming speech responses (chunked audio or `stream_format: "sse"`) are not
  supported yet. The endpoint always returns the complete audio file in a single
  response, so there is no low-latency, play-as-you-go output for now.
</Callout>

## Voices [#voices]

Gemini exposes 30 prebuilt voices. A few common ones:
`Kore`, `Puck`, `Zephyr`, `Charon`, `Fenrir`, `Leda`, `Orus`, `Aoede`. When
`voice` is omitted on a Gemini model, `Kore` is used.

OpenAI voices include `alloy`, `ash`, `ballad`, `coral`, `echo`, `fable`,
`nova`, `onyx`, `sage`, `shimmer`, and `verse`. When `voice` is omitted on an
OpenAI model, `alloy` is used.

ElevenLabs models accept 20 named voices, including `Sarah`, `Aria`, `Roger`,
`Laura`, `Charlie`, `George`, `Charlotte`, `Jessica`, `Brian`, and `Lily`. When
`voice` is omitted on an ElevenLabs model, `Sarah` is used. A raw ElevenLabs
voice id is also accepted directly.

Qwen-Audio-3.0-TTS voices are model-specific and cannot be mixed between
models: `longanlingxin` and `longanlufeng` on Plus (default `longanlingxin`),
and `longanhuan_v3.6`, `longjielidou_v3.6`, `loongeva_v3.6`, and `loongjohn` on
Flash (default `longanhuan_v3.6`).

## ElevenLabs [#elevenlabs]

The four ElevenLabs models are billed per **input character** (see the [models
page](https://llmgateway.io/models?filters=1\&audioGeneration=true) for rates):

* `eleven-multilingual-v2` — most lifelike, rich emotional expression, 29 languages
* `eleven-v3` — most expressive and human-like, 70+ languages
* `eleven-flash-v2-5` — ultra-low latency, 32 languages
* `eleven-turbo-v2-5` — fast and balanced, 32 languages

```bash
curl -X POST "https://api.llmgateway.io/v1/audio/speech" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "eleven-multilingual-v2",
    "input": "Hello, welcome to LLM Gateway!",
    "voice": "Sarah"
  }' \
  --output speech.mp3
```


# Request Timeouts
URL: https://docs.llmgateway.io/features/timeouts


The gateway enforces hard time limits on every request. These protect the
platform from stuck upstream connections, but they matter to you directly if
you run long generations or agentic pipelines: a single completion that runs
longer than the limit is terminated.

| Limit             | Default    | Applies to                                               |
| ----------------- | ---------- | -------------------------------------------------------- |
| **Streaming**     | 20 minutes | Streaming completions (`stream: true`)                   |
| **Non-streaming** | 10 minutes | Non-streaming completions                                |
| **End-to-end**    | 25 minutes | Overall ceiling for the whole request, including retries |

## How the limits behave [#how-the-limits-behave]

* **They are total-duration limits, not idle limits.** The timer starts when
  the gateway opens the upstream provider request and keeps running for the
  entire response — a stream is cut at the limit even while it is actively
  producing tokens.
* **They apply per request, not per session or conversation.** Every
  completion call gets a fresh window. A long-running agent that makes many
  calls over hours is unaffected; only a *single* call exceeding the limit
  fails.
* **Tool execution doesn't count.** When a completion finishes with
  `tool_calls`, the HTTP request ends. The time your application spends
  running tools (or coordinating subagents) between requests never consumes
  the window.
* **On timeout** the gateway returns a `504` with type `timeout_error` (see
  [Error Handling](https://docs.llmgateway.io/resources/error-handling)). If the limit is hit
  mid-stream, the stream terminates.

## Long-running agentic workloads [#long-running-agentic-workloads]

Coordinator/subagent architectures often hold a "master" completion open while
work happens elsewhere. To stay within the limits:

* **Keep any single completion under 20 minutes.** Structure the coordinator
  as a loop of discrete completions (the standard tool-calling pattern) rather
  than one long-lived request that spans the whole investigation.
* **Stream long generations.** Non-streaming requests are capped at 10
  minutes; streaming raises the per-request budget to 20 minutes and delivers
  partial output as it is produced.
* **Split very long outputs.** If a single generation legitimately needs more
  than 20 minutes (very large outputs on slow models, extensive reasoning),
  break it into continuation requests.

## Changing the limits [#changing-the-limits]

**Per-project overrides (Enterprise)** can be set under **Project Settings →
Routing** — see [Per-Project Routing
Configuration](https://docs.llmgateway.io/features/routing#per-project-routing-configuration-enterprise).
Overrides can only *lower* the timeouts: the defaults above are the
infrastructure ceiling on the hosted platform and cannot be raised per
project.

**Self-hosted deployments** control the limits with environment variables on
the gateway service:

| Variable                  | Default   | Controls                  |
| ------------------------- | --------- | ------------------------- |
| `AI_STREAMING_TIMEOUT_MS` | `1200000` | Streaming completions     |
| `AI_TIMEOUT_MS`           | `600000`  | Non-streaming completions |
| `GATEWAY_TIMEOUT_MS`      | `1500000` | End-to-end ceiling        |

<Callout type="warn">
  When raising the limits on a self-hosted deployment, raise the surrounding
  infrastructure in lockstep: your load balancer's backend/response timeout and
  the gateway's shutdown grace period (`SHUTDOWN_GRACE_PERIOD_MS`, and e.g.
  Kubernetes `terminationGracePeriodSeconds`) must all be at least as long as
  the longest stream you allow, or rollouts and intermediaries will still cut
  long requests.
</Callout>

<Callout type="info">
  If your workload genuinely needs single completions longer than 20 minutes on
  the hosted platform, contact us at [contact@llmgateway.io](mailto:contact@llmgateway.io) — the ceiling is an
  infrastructure setting, not a per-model constraint.
</Callout>


# Transcription
URL: https://docs.llmgateway.io/features/transcription


LLMGateway exposes a dedicated `/v1/audio/transcriptions` endpoint for
speech-to-text. It transcribes audio files into text with word-level
timestamps, optional speaker diarization, and inverse text normalization
(spoken numbers and currencies formatted in their written form).

Use it when you want to:

* Turn recordings, voicemails, or podcast episodes into text
* Generate captions or searchable transcripts with per-word timing
* Feed spoken content into a downstream model or RAG pipeline

For the full request and response schema, see the
[API reference](https://docs.llmgateway.io/v1_audio_transcriptions).

## Endpoint [#endpoint]

`POST https://api.llmgateway.io/v1/audio/transcriptions`

Authenticate with your LLMGateway API key:

```bash
-H "Authorization: Bearer $LLM_GATEWAY_API_KEY"
```

The current model is `grok-stt-1-0&#x60;, billed at **$0.10 per hour** of input
audio (against the duration reported by the provider).

## Parameters [#parameters]

The request body is `multipart/form-data`. Either `file` or `url` must be
provided.

| Parameter      | Type   | Default  | Description                                                                                               |
| -------------- | ------ | -------- | --------------------------------------------------------------------------------------------------------- |
| `model`        | string | required | The transcription model to use                                                                            |
| `file`         | file   | —        | The audio file to transcribe (WAV, MP3, OGG, Opus, FLAC, AAC, MP4, M4A, and more)                         |
| `url`          | string | —        | URL of an audio file to download and transcribe instead of uploading one                                  |
| `language`     | string | —        | Language code (e.g. `en`). When set, enables formatting of numbers and currencies into their written form |
| `diarize`      | string | `false`  | When `"true"`, each word in the response includes a `speaker` field identifying the detected speaker      |
| `filler_words` | string | `false`  | When `"true"`, filler words (e.g. "uh", "um") are kept in the transcript instead of being removed         |
| `keyterm`      | string | —        | A key term to bias transcription toward (e.g. product names). Repeat the field for multiple terms         |

## Response [#response]

The response includes the full transcript, audio duration, and word-level
timestamps:

```json
{
	"text": "The balance is $167,983.15.",
	"language": "English",
	"duration": 3.45,
	"words": [
		{ "text": "The", "start": 0.24, "end": 0.48 },
		{ "text": "balance", "start": 0.48, "end": 0.96 },
		{ "text": "is", "start": 0.96, "end": 1.12 },
		{ "text": "$167,983.15.", "start": 1.12, "end": 3.2 }
	]
}
```

## curl [#curl]

```bash
curl -X POST "https://api.llmgateway.io/v1/audio/transcriptions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -F model=grok-stt-1-0 \
  -F language=en \
  -F file=@audio.mp3
```

### Transcribing from a URL [#transcribing-from-a-url]

```bash
curl -X POST "https://api.llmgateway.io/v1/audio/transcriptions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -F model=grok-stt-1-0 \
  -F url="https://example.com/audio.mp3"
```


# Video Generation
URL: https://docs.llmgateway.io/features/video-generation


LLMGateway supports asynchronous video generation through an OpenAI-compatible `POST /v1/videos` flow.

Currently available models:

* **Veo 3.1** through `google-vertex` (720p, 1080p, 4k)
* **Seedance 2.5** through `bytedance` (480p, 720p, 1080p; clips up to 30 seconds)
* **Seedance 2.0** and **Seedance 1.5 Pro** through `bytedance` (720p, 1080p), **Seedance 2.0 Fast** through `bytedance` (720p only)
* **Seedance 2.0 Mini** through `bytedance` (480p, 720p)
* **KLING v3.0** and **KLING v3.0 Turbo** through `atlascloud` (720p, 1080p; KLING v3.0 also 4k)

You can find the current list of video-capable models on our [models page with the video filter enabled](https://llmgateway.io/models?filters=1\&videoGeneration=true) or programmatically through the [/v1/models endpoint](https://docs.llmgateway.io/v1_models).

## What Works Today [#what-works-today]

* `POST /v1/videos`
* `GET /v1/videos/{video_id}`
* `GET /v1/videos/{video_id}/content`
* Optional signed callbacks with `callback_url` and `callback_secret`

## Request Format [#request-format]

LLMGateway currently supports a focused subset of the OpenAI video API.

### Supported fields [#supported-fields]

| Field              | Type    | Required | Description                                                                                                                |
| ------------------ | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `model`            | string  | yes      | Any video-capable model from the filtered models page                                                                      |
| `prompt`           | string  | yes      | Text prompt for the video                                                                                                  |
| `seconds`          | number  | yes      | Duration in seconds. Supported values depend on the model (see below)                                                      |
| `size`             | string  | no       | `widthxheight`, limited to the sizes supported by the selected model and provider                                          |
| `audio`            | boolean | no       | Whether to include audio in the output (default `true`). Only honored when the model supports both audio and silent output |
| `image`            | object  | no       | Optional first frame for image-to-video generation (see first/last frame inputs below)                                     |
| `last_frame`       | object  | no       | Optional ending frame when `image` is provided (see first/last frame inputs below)                                         |
| `reference_images` | array   | no       | One to three provider-specific image inputs                                                                                |
| `input_reference`  | object  | no       | Alias for one or more `reference_images`                                                                                   |
| `reference_videos` | array   | no       | One to three reference video HTTPS URLs (Seedance 2.x only, see below)                                                     |
| `reference_audios` | array   | no       | One to three reference audio HTTPS URLs (Seedance 2.x only, see below)                                                     |
| `callback_url`     | string  | no       | LLMGateway extension for completion webhooks                                                                               |
| `callback_secret`  | string  | no       | LLMGateway extension used to sign webhook deliveries                                                                       |

### Sizes and durations by model [#sizes-and-durations-by-model]

| Model family      | Provider        | Supported sizes                                                                   | Supported durations |
| ----------------- | --------------- | --------------------------------------------------------------------------------- | ------------------- |
| Veo 3.1           | `google-vertex` | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`, `3840x2160`, `2160x3840`        | `4`, `6`, `8`, `10` |
| Seedance 2.5      | `bytedance`     | `848x480`, `854x480`, `480x854`, `1280x720`, `720x1280`, `1920x1080`, `1080x1920` | `4`–`30`            |
| Seedance 2.0      | `bytedance`     | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`                                  | `4`–`15`            |
| Seedance 2.0 Fast | `bytedance`     | `1280x720`, `720x1280`                                                            | `4`–`15`            |
| Seedance 2.0 Mini | `bytedance`     | `1280x720`, `720x1280`, `848x480`, `854x480`, `480x854`                           | `4`–`15`            |
| Seedance 1.5 Pro  | `bytedance`     | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`                                  | `5`, `10`           |
| KLING v3.0        | `atlascloud`    | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`, `3840x2160`, `2160x3840`        | `5`, `10`           |
| KLING v3.0 Turbo  | `atlascloud`    | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`                                  | `5`, `10`           |

Requests return `400` when the selected provider cannot serve the requested `size` or `seconds`. Seedance and KLING v3.0 derive `aspect_ratio` from the requested `size` (16:9 for landscape, 9:16 for portrait).

**KLING v3.0 Turbo** always generates audio and does not support `audio: false`; silent requests return a `400`. Use **KLING v3.0** (`kling-v3-0`) when you need silent output.

### First/last frame inputs [#firstlast-frame-inputs]

Frame inputs interpolate a video between a starting frame and an optional ending frame. You provide the first frame as `image` and, optionally, the ending frame as `last_frame`. The gateway tags each one with the correct role for the provider, so you don't set roles yourself.

| Field        | Required           | Accepted input                   | Available on                                                                                                       |
| ------------ | ------------------ | -------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `image`      | for frame mode     | HTTPS URL **or** base64 data URL | **Seedance 2.x** (`bytedance`), **KLING v3.0 / Turbo** (`atlascloud`), Veo 3.1 (`google-vertex`), `minimax`, `xai` |
| `last_frame` | no (needs `image`) | HTTPS URL **or** base64 data URL | **Seedance 2.x** (`bytedance`), **KLING v3.0 / Turbo** (`atlascloud`), Veo 3.1 (`google-vertex`)                   |

#### Rules and limits [#rules-and-limits]

* **Seedance scope.** Frame inputs are supported on **Seedance 2.5**, **Seedance 2.0**, **Seedance 2.0 Fast**, and **Seedance 2.0 Mini** (`seedance-2-5`, `seedance-2-0`, `seedance-2-0-fast`, `seedance-2-0-mini`). Sending `image`/`last_frame` to Seedance 1.5 Pro or any other ByteDance model returns a `400`.
* **KLING scope.** Frame inputs are supported on **KLING v3.0** and **KLING v3.0 Turbo** (`kling-v3-0`, `kling-v3-0-turbo`). The gateway uploads base64 frames to AtlasCloud's media endpoint automatically, so both HTTPS URLs and base64 data URLs are accepted.
* **`last_frame` requires `image`.** Providing `last_frame` without `image` returns a `400`.
* **Not combinable with references.** First/last frame inputs (`image`, `last_frame`) cannot be combined with reference inputs (`reference_images`, `input_reference`, `reference_videos`, `reference_audios`).

#### Example (Seedance 2.0) [#example-seedance-20]

```bash
curl -X POST "https://api.llmgateway.io/v1/videos" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "Morph smoothly from the first frame into the last frame",
    "seconds": 5,
    "size": "1280x720",
    "image": { "image_url": "https://example.com/first-frame.png" },
    "last_frame": { "image_url": "https://example.com/last-frame.png" }
  }'
```

#### Example (KLING v3.0 image-to-video) [#example-kling-v30-image-to-video]

```bash
curl -X POST "https://api.llmgateway.io/v1/videos" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kling-v3-0",
    "prompt": "Animate the scene with gentle camera motion",
    "seconds": 5,
    "size": "1280x720",
    "image": { "image_url": "https://example.com/first-frame.png" }
  }'
```

### Reference-guided generation (Seedance 2.x) [#reference-guided-generation-seedance-2x]

Seedance 2.x (`seedance-2-5`, `seedance-2-0`, `seedance-2-0-fast`, `seedance-2-0-mini`) can generate a video that is guided by reference **images**, **videos**, and **audio** — sometimes called omni-reference. You attach references as top-level fields in the same `POST /v1/videos` payload; the gateway forwards each one to the provider tagged with the correct role, so you don't set roles yourself.

| Reference type | Payload field                                | Count | Accepted input                   | Available on                            |
| -------------- | -------------------------------------------- | ----- | -------------------------------- | --------------------------------------- |
| Image          | `reference_images` (`input_reference` alias) | 1–3   | HTTPS URL **or** base64 data URL | Seedance 2.x, Veo 3.1 (`google-vertex`) |
| Video          | `reference_videos`                           | 1–3   | HTTPS URL only                   | Seedance 2.x                            |
| Audio          | `reference_audios`                           | 1–3   | HTTPS URL only                   | Seedance 2.x                            |

Each list item accepts either a bare URL string or an object form:

* `reference_images`: `"https://…/subject.png"` or `{ "image_url": "https://…/subject.png" }`
* `reference_videos`: `"https://…/motion.mp4"` or `{ "video_url": "https://…/motion.mp4" }`
* `reference_audios`: `"https://…/track.mp3"` or `{ "audio_url": "https://…/track.mp3" }`

You can mix all three reference types in one request. The `prompt` can be a light instruction (for example `"adapt this to show more detail"`) — the references drive the result.

#### Rules and limits [#rules-and-limits-1]

* **HTTPS only for video and audio.** `reference_videos` and `reference_audios` must be publicly reachable HTTPS URLs (the provider fetches them). base64 data URLs are rejected for video/audio; images may be HTTPS URLs or base64 data URLs.
* **Reference video resolution.** Seedance requires reference video frames to be at least \~409,600 pixels (roughly 480p or larger). Low-resolution clips such as 360p are rejected with a `400`.
* **Not combinable with frames.** Reference inputs (`reference_images`, `reference_videos`, `reference_audios`) cannot be combined with the first/last frame inputs (`image`, `last_frame`).
* **Provider scope.** Reference videos and audio are only supported on Seedance 2.x models; sending them to other models returns a `400`. **KLING v3.0** does not support any reference inputs (`reference_images`, `reference_videos`, `reference_audios`); use first/last frame inputs instead.
* **Moderation still applies.** The output is subject to the provider's content moderation. Blocked generations finish as `failed` and are logged with a `content_filter` finish reason.

#### Examples [#examples]

Reference images only (subjects / style):

```bash
curl -X POST "https://api.llmgateway.io/v1/videos" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "The subject walks through a neon-lit market at night",
    "seconds": 5,
    "size": "1280x720",
    "reference_images": [
      { "image_url": "https://example.com/subject.png" },
      { "image_url": "https://example.com/style.png" }
    ]
  }'
```

Reference video only (motion / scene — let the clip drive the output):

```bash
curl -X POST "https://api.llmgateway.io/v1/videos" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "adapt this to show more detail",
    "seconds": 5,
    "size": "1280x720",
    "reference_videos": ["https://example.com/reference-motion.mp4"]
  }'
```

All three reference types combined:

```bash
curl -X POST "https://api.llmgateway.io/v1/videos" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "The subject performs the choreography from the reference video",
    "seconds": 5,
    "size": "1280x720",
    "reference_images": [
      { "image_url": "https://example.com/subject.png" }
    ],
    "reference_videos": [
      "https://example.com/reference-motion.mp4"
    ],
    "reference_audios": [
      "https://example.com/reference-track.mp3"
    ]
  }'
```

### Not supported yet [#not-supported-yet]

* multipart uploads
* `n` values other than `1`
* remix/list/delete video endpoints

## Create a Video [#create-a-video]

Video generation requires at least `$1.00` in available organization credits before the job is submitted upstream.

Pricing is per second of generated video. For Seedance and KLING v3.0, enabling audio can increase the per-second rate on models that price audio and video separately.

Veo 3.1:

| Model                           | Provider        | Supported sizes                                  | Price            |
| ------------------------------- | --------------- | ------------------------------------------------ | ---------------- |
| `veo-3.1-generate-preview`      | `google-vertex` | `1280x720`, `720x1280`, `1920x1080`, `1080x1920` | `$0.40 / second` |
| `veo-3.1-fast-generate-preview` | `google-vertex` | `1280x720`, `720x1280`, `1920x1080`, `1080x1920` | `$0.15 / second` |
| `veo-3.1-generate-preview`      | `google-vertex` | `3840x2160`, `2160x3840`                         | `$0.60 / second` |
| `veo-3.1-fast-generate-preview` | `google-vertex` | `3840x2160`, `2160x3840`                         | `$0.35 / second` |

Seedance (ByteDance):

| Model               | Provider    | Resolution | With audio          | Video only          |
| ------------------- | ----------- | ---------- | ------------------- | ------------------- |
| `seedance-2-5`      | `bytedance` | 480p       | `$0.1028 / second`  | `$0.1028 / second`  |
| `seedance-2-5`      | `bytedance` | 720p       | `$0.2311 / second`  | `$0.2311 / second`  |
| `seedance-2-5`      | `bytedance` | 1080p      | `$0.52 / second`    | `$0.52 / second`    |
| `seedance-2-0`      | `bytedance` | 720p       | `$0.1512 / second`  | `$0.1512 / second`  |
| `seedance-2-0`      | `bytedance` | 1080p      | `$0.3402 / second`  | `$0.3402 / second`  |
| `seedance-2-0-fast` | `bytedance` | 720p       | `$0.121 / second`   | `$0.121 / second`   |
| `seedance-2-0-mini` | `bytedance` | 480p       | `$0.0378 / second`  | `$0.0378 / second`  |
| `seedance-2-0-mini` | `bytedance` | 720p       | `$0.0756 / second`  | `$0.0756 / second`  |
| `seedance-1-5-pro`  | `bytedance` | 720p       | `$0.05184 / second` | `$0.02592 / second` |
| `seedance-1-5-pro`  | `bytedance` | 1080p      | `$0.1166 / second`  | `$0.05832 / second` |

KLING (AtlasCloud):

| Model              | Provider     | Resolution | With audio        | Video only        |
| ------------------ | ------------ | ---------- | ----------------- | ----------------- |
| `kling-v3-0`       | `atlascloud` | 720p       | `$0.126 / second` | `$0.084 / second` |
| `kling-v3-0`       | `atlascloud` | 1080p      | `$0.168 / second` | `$0.112 / second` |
| `kling-v3-0`       | `atlascloud` | 4k         | `$0.42 / second`  | `$0.42 / second`  |
| `kling-v3-0-turbo` | `atlascloud` | 720p       | `$0.168 / second` | n/a (audio only)  |
| `kling-v3-0-turbo` | `atlascloud` | 1080p      | `$0.21 / second`  | n/a (audio only)  |

```bash
curl -X POST "https://api.llmgateway.io/v1/videos" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "veo-3.1-generate-preview",
    "prompt": "A cinematic aerial shot flying above a rainforest waterfall at sunrise",
    "seconds": 8,
    "size": "1920x1080"
  }'
```

Example response:

```json
{
	"id": "v_123",
	"object": "video",
	"model": "veo-3.1-generate-preview",
	"status": "queued",
	"progress": 0,
	"created_at": 1773600000,
	"completed_at": null,
	"expires_at": null,
	"error": null
}
```

## Retrieve Job Status [#retrieve-job-status]

```bash
curl "https://api.llmgateway.io/v1/videos/v_123" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY"
```

Typical statuses:

* `queued`
* `in_progress`
* `completed`
* `failed`
* `canceled`
* `expired`

`google-vertex` follows Vertex AI's long-running operation flow. The gateway submits Veo generation with `predictLongRunning`, polls with `fetchPredictOperation`, and streams the final bytes through the gateway content endpoint once the operation is done.

`bytedance` uses the ModelArk `/contents/generations/tasks` endpoint. The gateway submits the job, polls the upstream task status, and exposes the final video bytes through the gateway content endpoint once the task succeeds.

`atlascloud` uses the AtlasCloud `/api/v1/model/generateVideo` endpoint and polls `/api/v1/model/prediction/{id}` for status. The gateway resolves the upstream KLING variant (standard, turbo, or 4k) and task type (text-to-video or image-to-video) from your request, then streams the final video bytes through the gateway content endpoint once the prediction completes.

## Download the Video [#download-the-video]

Once the job is complete, stream the resulting video bytes from the content endpoint:

```bash
curl "https://api.llmgateway.io/v1/videos/v_123/content" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  --output video.mp4
```

## Signed Callbacks [#signed-callbacks]

LLMGateway can notify your application when the job reaches a terminal state.

```bash
curl -X POST "https://api.llmgateway.io/v1/videos" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "veo-3.1-fast-generate-preview",
    "prompt": "A slow-motion close-up of waves crashing against black volcanic rock",
    "seconds": 8,
    "callback_url": "https://example.com/webhooks/video",
    "callback_secret": "whsec_your_secret_here"
  }'
```

### Delivery behavior [#delivery-behavior]

* Callbacks are sent only for terminal states in v1
* Event types are `video.completed` and `video.failed`
* Deliveries retry with exponential backoff on network errors, timeouts, and non-2xx responses
* Each attempt is recorded internally in the webhook delivery log table

### Headers [#headers]

* `webhook-id`
* `webhook-timestamp`
* `webhook-signature`

### Signature format [#signature-format]

LLMGateway signs the string:

```text
{webhook-id}.{webhook-timestamp}.{raw-request-body}
```

using HMAC-SHA256 with your `callback_secret`, then sends:

```text
webhook-signature: v1,{base64_signature}
```

### Verification example [#verification-example]

```ts
import { createHmac, timingSafeEqual } from "node:crypto";

function verifyWebhook(
	body: string,
	webhookId: string,
	webhookTimestamp: string,
	webhookSignature: string,
	secret: string,
) {
	const expected = createHmac("sha256", secret)
		.update(`${webhookId}.${webhookTimestamp}.${body}`)
		.digest("base64");

	const provided = webhookSignature.replace(/^v1,/, "");

	return timingSafeEqual(Buffer.from(expected), Buffer.from(provided));
}
```

## Related Docs [#related-docs]

* [Image Generation](https://docs.llmgateway.io/features/image-generation)
* [Routing](https://docs.llmgateway.io/features/routing)
* [Models API](https://docs.llmgateway.io/v1_models)


# Vision Support
URL: https://docs.llmgateway.io/features/vision


LLMGateway supports vision-enabled models that can analyze and describe images. You can provide images via HTTPS URLs or inline base64-encoded data.

## Vision-Enabled Models [#vision-enabled-models]

You can find all vision-enabled models on our [models page with vision filter](https://llmgateway.io/models?filters=1\&vision=true). These models can process both text and image content in the same request.

## Image Formats [#image-formats]

### Using HTTPS URLs [#using-https-urls]

You can provide any publicly accessible HTTPS URL pointing to an image:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What do you see in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://example.com/image.jpg"
            }
          }
        ]
      }
    ]
  }'
```

### Using Base64 Inline Data [#using-base64-inline-data]

You can also provide images as base64-encoded data URIs:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this image"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD..."
            }
          }
        ]
      }
    ]
  }'
```

## Content Array Format [#content-array-format]

When using vision models, the `content` field should be an array containing both text and image content blocks:

* **Text content**: `{"type": "text", "text": "Your message"}`
* **Image content**: `{"type": "image_url", "image_url": {"url": "image_url_or_data_uri"}}`

## Multiple Images [#multiple-images]

You can include multiple images in a single request:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Compare these two images"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://example.com/image1.jpg"
            }
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://example.com/image2.jpg"
            }
          }
        ]
      }
    ]
  }'
```

## Simple String Content [#simple-string-content]

<Callout type="info">
  For vision models, you can still use simple string content for text-only
  messages. The array format is only required when including images.
</Callout>

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello! How can you help me today?"
      }
    ]
  }'
```

## Supported Image Types [#supported-image-types]

Vision models typically support common image formats including:

* JPEG (.jpg, .jpeg)
* PNG (.png)
* WebP (.webp)
* GIF (.gif)

The specific formats supported may vary by model provider. Check the individual model documentation for format limitations and file size restrictions.

## Error Handling [#error-handling]

If an image URL is inaccessible or the image format is unsupported, the gateway will handle the error gracefully and may substitute a placeholder or error message in the request to the underlying model.


# Native Web Search
URL: https://docs.llmgateway.io/features/web-search


LLM Gateway supports native web search capabilities that allow models to access real-time information from the internet. This feature is useful for answering questions about current events, recent news, live data, and other time-sensitive information that may not be in the model's training data.

## How It Works [#how-it-works]

When you include the `web_search` tool in your request, the model can search the web to gather relevant information before generating a response:

1. You send a request with the `web_search` tool enabled
2. The model determines if web search is needed based on the query, unless you
   [require one](#requiring-a-search)
3. If needed, the model performs web searches to gather current information
4. The model synthesizes the search results and generates a response
5. Citations are included in the response to show information sources

## Supported Providers [#supported-providers]

Native web search is available on select models. See all models with native web search support on our [models page](https://llmgateway.io/models?filters=1\&webSearch=true).

## Basic Usage [#basic-usage]

To enable web search, add the `web_search` tool to your request:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.2",
    "messages": [
      {
        "role": "user",
        "content": "What is the current weather in San Francisco?"
      }
    ],
    "tools": [
      {
        "type": "web_search"
      }
    ]
  }'
```

### Example Response [#example-response]

```json
{
	"id": "chatcmpl-abc123",
	"object": "chat.completion",
	"created": 1234567890,
	"model": "openai/gpt-5.2",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "The current weather in San Francisco is 57°F (14°C) with mostly cloudy skies...",
				"annotations": [
					{
						"type": "url_citation",
						"url": "https://weather.com/...",
						"title": "San Francisco Weather"
					}
				]
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 15,
		"completion_tokens": 150,
		"total_tokens": 165,
		"cost": 0.0315
	}
}
```

## Web Search Options [#web-search-options]

The `web_search` tool accepts optional configuration parameters:

### User Location [#user-location]

Provide location context to get more relevant local search results:

```json
{
	"type": "web_search",
	"user_location": {
		"city": "San Francisco",
		"region": "California",
		"country": "US",
		"timezone": "America/Los_Angeles"
	}
}
```

### Search Context Size [#search-context-size]

Control the amount of web content retrieved (OpenAI only):

```json
{
	"type": "web_search",
	"search_context_size": "medium"
}
```

Available values:

* `low` - Minimal search context, faster responses
* `medium` - Balanced context (default)
* `high` - Maximum search context, more comprehensive

### Max Uses [#max-uses]

Limit the number of searches per request (provider-dependent):

```json
{
	"type": "web_search",
	"max_uses": 3
}
```

## Requiring a Search [#requiring-a-search]

By default the `web_search` tool offers the model a search and lets it judge
whether the question needs one. To require a search on every request, set
`tool_choice`:

```json
{
	"model": "...",
	"messages": [{ "role": "user", "content": "What shipped in AI this week?" }],
	"tools": [{ "type": "web_search" }],
	"tool_choice": { "type": "web_search" }
}
```

Reach for this sparingly. A forced search is billed on every request that
carries it, and the retrieved snippets are appended to your prompt, so they are
billed as input tokens too — on a question the model could have answered from
memory, you pay for both and gain nothing. If you are building a chat interface
with a "web search" toggle, leaving the tool attached with the default
`tool_choice` is usually what you want, so that follow-ups like "shorter,
please" do not trigger a search.

A few upstreams have no model-elected search at all and can only search when
asked to. Requiring a search is the only way to use their search; without it
they behave like a model that decided not to search, and the gateway prefers to
route a merely offered tool to a provider that can make that decision for
itself. You can find the models with native web search on the
[models page](https://llmgateway.io/models?filters=1\&webSearch=true).

## Using with SDKs [#using-with-sdks]

### OpenAI SDK (Python) [#openai-sdk-python]

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://api.llmgateway.io/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="gpt-5.2",
    messages=[
        {"role": "user", "content": "What are the latest news headlines today?"}
    ],
    tools=[{"type": "web_search"}]
)

print(response.choices[0].message.content)
```

### OpenAI SDK (TypeScript) [#openai-sdk-typescript]

```typescript
import OpenAI from "openai";

const client = new OpenAI({
	baseURL: "https://api.llmgateway.io/v1",
	apiKey: "your-api-key",
});

const response = await client.chat.completions.create({
	model: "gpt-5.2",
	messages: [{ role: "user", content: "What are the latest tech news?" }],
	tools: [{ type: "web_search" }],
});

console.log(response.choices[0].message.content);
```

## Streaming [#streaming]

Web search works with streaming responses. Citations are included in the final chunks:

```bash
curl -X POST "https://api.llmgateway.io/v1/chat/completions" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.2",
    "messages": [
      {"role": "user", "content": "What is the current stock price of Apple?"}
    ],
    "tools": [{"type": "web_search"}],
    "stream": true
  }'
```

## Citations and Sources [#citations-and-sources]

Web search responses include citations to show where information was sourced from. These appear in the `annotations` field of the message:

```json
{
	"annotations": [
		{
			"type": "url_citation",
			"url": "https://example.com/article",
			"title": "Article Title",
			"start_index": 0,
			"end_index": 50
		}
	]
}
```

<Callout type="info">
  Citation format may vary slightly between providers, but LLM Gateway
  normalizes them into a consistent structure.
</Callout>

## Cost Tracking [#cost-tracking]

Web search costs are rolled into the total `cost` reported in the usage object:

```json
{
	"usage": {
		"prompt_tokens": 15,
		"completion_tokens": 150,
		"total_tokens": 165,
		"cost": 0.0125,
		"cost_details": {
			"upstream_inference_cost": 0.0115,
			"upstream_inference_prompt_cost": 0.0015,
			"upstream_inference_completions_cost": 0.01,
			"total_cost": 0.0125,
			"input_cost": 0.0015,
			"output_cost": 0.01,
			"web_search_cost": 0.001
		}
	}
}
```

Web search is billed at $0.01 per search call for reasoning models (GPT-5, o-series) and $0.025 per call for non-reasoning models. The web search charge is included in the top-level `cost` value and surfaced separately as `cost_details.web_search_cost`.

## Combining with Function Tools [#combining-with-function-tools]

You can use web search alongside regular function tools:

```json
{
	"tools": [
		{ "type": "web_search" },
		{
			"type": "function",
			"function": {
				"name": "get_weather",
				"description": "Get weather for a location",
				"parameters": {
					"type": "object",
					"properties": {
						"location": { "type": "string" }
					}
				}
			}
		}
	]
}
```

<Callout type="warning">
  Some dedicated search models only support web search and do not support
  additional function tools. Use `gpt-5.2` or other GPT-5 series models if you
  need both web search and function tools.
</Callout>

## Use Cases [#use-cases]

### Current Events and News [#current-events-and-news]

```json
{
	"messages": [
		{ "role": "user", "content": "What are the major news stories today?" }
	],
	"tools": [{ "type": "web_search" }]
}
```

### Real-Time Data [#real-time-data]

```json
{
	"messages": [
		{ "role": "user", "content": "What is the current price of Bitcoin?" }
	],
	"tools": [{ "type": "web_search" }]
}
```

### Research and Fact-Checking [#research-and-fact-checking]

```json
{
	"messages": [
		{
			"role": "user",
			"content": "What are the latest findings on climate change?"
		}
	],
	"tools": [{ "type": "web_search" }]
}
```

### Local Information [#local-information]

```json
{
	"messages": [
		{
			"role": "user",
			"content": "What restaurants are open near me right now?"
		}
	],
	"tools": [
		{
			"type": "web_search",
			"user_location": {
				"city": "New York",
				"country": "US"
			}
		}
	]
}
```

## Best Practices [#best-practices]

1. **Use GPT-5.2**: For the best web search experience with full tool support, use `gpt-5.2`
2. **Provide location context**: When queries are location-dependent, include `user_location` for more relevant results
3. **Monitor costs**: Web search incurs per-query costs in addition to token costs
4. **Check citations**: Always review the citations in responses to verify information sources
5. **Use streaming**: For user-facing applications, enable streaming to show responses as they're generated

## Error Handling [#error-handling]

If you try to use web search with a model that doesn't support it:

```json
{
	"error": {
		"message": "Model gpt-4o does not support native web search. Remove the web_search tool or use a model that supports it. See https://llmgateway.io/models?features=webSearch for supported models.",
		"type": "invalid_request_error"
	}
}
```

To avoid this error, only use the `web_search` tool with [native web search enabled models](https://llmgateway.io/models?filters=1\&webSearch=true).


# Agent Skills
URL: https://docs.llmgateway.io/guides/agent-skills


**Agent Skills** are structured guidelines for AI coding agents, optimized for use with LLM Gateway and the AI SDK. They provide best practices and reusable instructions that help AI agents generate higher-quality code.

## What Are Agent Skills? [#what-are-agent-skills]

Agent Skills are packaged sets of rules and guidelines that teach AI coding agents how to implement specific features correctly. Each skill covers:

* API integration patterns
* Frontend rendering best practices
* Error handling strategies
* Performance optimization techniques

## Available Skills [#available-skills]

### Image Generation [#image-generation]

The Image Generation skill teaches AI agents how to properly implement image generation features:

* **API Integration** — correctly calling image generation APIs
* **Frontend Rendering** — displaying generated images efficiently
* **Error Handling** — graceful degradation and retry logic
* **Performance** — caching, lazy loading, and optimization

### Changelog Writer [#changelog-writer]

The Changelog Writer skill drafts a polished LLM Gateway changelog entry and its OpenGraph image in one pass:

* **House style** — writes in the established changelog voice: problem first, benefits over features, plain and confident
* **Correct format** — generates the dated Markdown file with valid frontmatter (`id`, `slug`, `date`, `title`, `summary`, `image`)
* **OG image prompt** — hands back a ready-to-run `gpt-image-2` prompt at the recommended **1536×1024** OpenGraph resolution
* **Validation** — formats and builds the entry before you commit

It ships as a [Claude Code skill](https://docs.claude.com/en/docs/claude-code/skills) — a single `SKILL.md` file. Trigger it by typing `changelog` (or "write a changelog entry") in Claude Code.

## Claude Code Skills [#claude-code-skills]

Some skills ship as &#x2A;*[Claude Code skills](https://docs.claude.com/en/docs/claude-code/skills)**: a single `SKILL.md` file with YAML frontmatter that Claude Code loads automatically when its description matches what you ask. The **Changelog Writer** above is one of them.

Install one by copying its folder into your project's `.claude/skills/` directory:

```bash
# from the llmgateway-templates repo
cp -r skills/changelog /path/to/your/project/.claude/skills/
```

Then trigger it by name inside Claude Code:

```
> changelog
```

Place the folder in `~/.claude/skills/` instead to make the skill available across every project. Browse the available Claude Code skills in the [llmgateway-templates repository](https://github.com/theopenco/llmgateway-templates/tree/main/skills).

## Installation [#installation]

<Steps>
  <Step>
    ### Prerequisites [#prerequisites]

    Ensure you have Node.js 18+ and pnpm 9+ installed:

    ```bash
    node --version  # v18.0.0 or higher
    pnpm --version  # 9.0.0 or higher
    ```
  </Step>

  <Step>
    ### Clone the Repository [#clone-the-repository]

    ```bash
    git clone https://github.com/theopenco/agent-skills.git
    cd agent-skills
    ```
  </Step>

  <Step>
    ### Install Dependencies [#install-dependencies]

    ```bash
    pnpm install
    ```
  </Step>

  <Step>
    ### Build Skills [#build-skills]

    Build all skills to generate the documentation:

    ```bash
    pnpm build:all
    ```

    Or build a specific skill:

    ```bash
    pnpm build
    ```
  </Step>
</Steps>

## Using Skills in Your Project [#using-skills-in-your-project]

After building, each skill generates an `AGENTS.md` file that can be used with AI coding agents like Claude, Cursor, or Copilot.

### With Claude Code [#with-claude-code]

Add the generated `AGENTS.md` content to your project's `CLAUDE.md` file:

```bash
cat skills/image-generation/AGENTS.md >> CLAUDE.md
```

### With Cursor [#with-cursor]

Add the skill content to your `.cursorrules` file:

```bash
cat skills/image-generation/AGENTS.md >> .cursorrules
```

### With Other AI Agents [#with-other-ai-agents]

Most AI coding tools support custom instructions. Copy the skill content into your tool's configuration.

## Project Structure [#project-structure]

```
agent-skills/
├── packages/
│   └── skills-build/          # Build tooling
├── skills/
│   └── image-generation/      # Individual skill
│       ├── rules/             # Rule files
│       ├── AGENTS.md          # Generated documentation
│       └── metadata.json      # Skill metadata
└── package.json
```

## Contributing [#contributing]

### Adding New Rules [#adding-new-rules]

<Steps>
  <Step>
    ### Fork and Clone [#fork-and-clone]

    Fork the repository and create a feature branch:

    ```bash
    git checkout -b feat/new-rule
    ```
  </Step>

  <Step>
    ### Create a Rule File [#create-a-rule-file]

    Rules follow a standardized template with YAML frontmatter containing `title`, `impact` (high/medium/low), and `tags`. The body includes sections for Context, Incorrect examples, and Correct examples with TypeScript code blocks.

    See existing rules in `skills/image-generation/rules/` for reference.
  </Step>

  <Step>
    ### Validate and Build [#validate-and-build]

    ```bash
    pnpm validate
    pnpm build:all
    ```
  </Step>

  <Step>
    ### Submit a Pull Request [#submit-a-pull-request]

    Push your changes and open a PR.
  </Step>
</Steps>

### Impact Levels [#impact-levels]

When creating rules, use these impact levels:

* **high** — Critical for correctness or security
* **medium** — Important for quality and maintainability
* **low** — Nice-to-have improvements

## Development Commands [#development-commands]

| Command          | Description                 |
| ---------------- | --------------------------- |
| `pnpm install`   | Install dependencies        |
| `pnpm build:all` | Build all skills            |
| `pnpm build`     | Build a specific skill      |
| `pnpm validate`  | Validate rule files         |
| `pnpm dev`       | Development mode with watch |

## More Resources [#more-resources]

* [GitHub Repository](https://github.com/theopenco/agent-skills) — Source code and contributions
* [LLM Gateway CLI](https://docs.llmgateway.io/developers/cli) — Project scaffolding tool
* [Templates](https://llmgateway.io/templates) — Production-ready starter projects

<Callout type="info">
  Want to contribute a new skill or rule? Check out the [contribution
  guidelines](https://github.com/theopenco/agent-skills#contributing) on GitHub.
</Callout>


# Autohand Code Integration
URL: https://llmgateway.io/guides/autohand


Autohand Code is an autonomous AI coding agent that works in your terminal, IDE, and Slack. With LLM Gateway, you can route all Autohand Code requests through a single gateway—use any of 200+ models from 40+ providers, with full cost tracking and smart routing.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for LLM Gateway [#sign-up-for-llm-gateway]

    [Sign up free](https://llmgateway.io/signup) — no credit card required. Create an API key in the dashboard and copy it when shown.
  </Step>

  <Step>
    ### Set Environment Variables [#set-environment-variables]

    Configure Autohand Code to use LLM Gateway:

    ```bash
    export OPENAI_BASE_URL=https://api.llmgateway.io/v1
    export OPENAI_API_KEY=llmgtwy_your_api_key_here
    ```
  </Step>

  <Step>
    ### Run Autohand Code [#run-autohand-code]

    ```bash
    autohand
    ```

    All requests will now be routed through LLM Gateway.
  </Step>
</Steps>

## Why Use LLM Gateway with Autohand Code [#why-use-llm-gateway-with-autohand-code]

* **200+ models** — GPT-5, Claude Opus, Gemini, Llama, and more from 40+ providers
* **Smart routing** — Automatically selects the best provider based on uptime, throughput, price, and latency
* **Cost tracking** — Monitor exactly how much each autonomous agent costs
* **Single bill** — No need to manage multiple API provider accounts
* **Response caching** — Repeated requests hit cache automatically
* **Automatic failover** — If one provider is down, requests route to another

## Configuration File [#configuration-file]

You can also configure LLM Gateway in Autohand Code's config file:

```json
{
	"provider": {
		"llmgateway": {
			"baseUrl": "https://api.llmgateway.io/v1",
			"apiKey": "llmgtwy_your_api_key_here"
		}
	},
	"model": "gpt-5"
}
```

## Choosing Models [#choosing-models]

You can use any model from the [models page](https://llmgateway.io/models).

| Model                    | Best For                                    |
| ------------------------ | ------------------------------------------- |
| `gpt-5`                  | Latest OpenAI flagship, highest quality     |
| `claude-opus-4-6`        | Anthropic's most capable model              |
| `claude-sonnet-4-6`      | Fast reasoning with extended thinking       |
| `gemini-3.1-pro-preview` | Google's latest flagship, 1M context window |
| `o3`                     | Advanced reasoning tasks                    |
| `gpt-5-mini`             | Cost-effective, quick responses             |
| `gemini-3.6-flash`       | Fast responses, good for high-volume        |
| `deepseek-v3.1`          | Open-source with vision and tools           |

## Autohand Code Features with LLM Gateway [#autohand-code-features-with-llm-gateway]

### Terminal (CLI) [#terminal-cli]

Autohand Code CLI works seamlessly with LLM Gateway. Set the environment variables and use all Autohand Code commands as normal—multi-file editing, agentic search, and autonomous code generation all work out of the box.

### IDE Integration [#ide-integration]

Autohand Code's VS Code and Zed extensions respect the same environment variables. Set them in your shell profile and the IDE integration will automatically route through LLM Gateway.

### Slack Integration [#slack-integration]

When using Autohand Code through Slack, configure the LLM Gateway base URL in your Autohand Code server settings to route all Slack-triggered coding tasks through the gateway.

## Monitoring Usage [#monitoring-usage]

Once configured, all Autohand Code requests appear in your LLM Gateway dashboard:

* **Request logs** — See every prompt and response
* **Cost breakdown** — Track spending by model and time period
* **Usage analytics** — Understand your AI usage patterns

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Claude Code Integration
URL: https://llmgateway.io/guides/claude-code


Claude Code is locked to Anthropic's API by default. With LLM Gateway, you can point it at any model—GPT-5, Gemini, Llama, or 180+ others—while keeping the same Anthropic API format Claude Code expects.

Three environment variables. No code changes. Full cost tracking in your dashboard.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for LLM Gateway [#sign-up-for-llm-gateway]

    [Sign up free](https://llmgateway.io/signup) — no credit card required. Create an API key in the dashboard and copy it when shown.
  </Step>

  <Step>
    ### Set Environment Variables [#set-environment-variables]

    Configure Claude Code to use LLM Gateway:

    ```bash
    export ANTHROPIC_BASE_URL=https://api.llmgateway.io
    export ANTHROPIC_AUTH_TOKEN=llmgtwy_your_api_key_here
    # optional: specify a model, otherwise it uses the default Claude model
    export ANTHROPIC_MODEL=gpt-5  # or any model from our catalog
    ```
  </Step>

  <Step>
    ### Run Claude Code [#run-claude-code]

    ```bash
    claude
    ```

    All requests will now be routed through LLM Gateway.
  </Step>
</Steps>

## Why This Works [#why-this-works]

LLM Gateway's `/v1/messages` endpoint speaks Anthropic's API format natively. We handle the translation to each provider behind the scenes. This means:

* **Use any model** — GPT-5, Gemini, Llama, or Claude itself
* **Keep your workflow** — Claude Code doesn't know the difference
* **Track costs** — Every request appears in your LLM Gateway dashboard
* **Automatic caching** — Repeated requests hit cache, saving money

## Choosing Models [#choosing-models]

You can use any model from the [models page](https://llmgateway.io/models).

### Use OpenAI's Latest Models [#use-openais-latest-models]

```bash
# Use the latest GPT model
export ANTHROPIC_MODEL=gpt-5

# Use a cost-effective alternative
export ANTHROPIC_MODEL=gpt-5-mini
```

### Use Google's Gemini [#use-googles-gemini]

```bash
export ANTHROPIC_MODEL=gemini-3.1-pro-preview
```

### Use Anthropic's Claude Models [#use-anthropics-claude-models]

```bash
export ANTHROPIC_MODEL=anthropic/claude-3-5-sonnet-20241022
```

## Predefining Models in a Settings File [#predefining-models-in-a-settings-file]

Environment variables are read once at startup, so changing `ANTHROPIC_MODEL` means restarting Claude Code. To switch models mid-session instead, put the configuration in `~/.claude/settings.json` (user-wide) or `.claude/settings.json` (per project) and use `/model` to switch on the fly:

```json title="~/.claude/settings.json"
{
	"env": {
		"ANTHROPIC_BASE_URL": "https://api.llmgateway.io",
		"ANTHROPIC_AUTH_TOKEN": "llmgtwy_your_api_key_here"
	},
	"model": "claude-sonnet-5"
}
```

`model` sets the model a new session starts on. `/model` overrides it for the running session and saves your choice as the new default.

### Restricting the Model List [#restricting-the-model-list]

`availableModels` limits which models the `/model` picker offers — useful for keeping a team on an approved, budget-appropriate set:

```json title="~/.claude/settings.json"
{
	"availableModels": ["claude-sonnet-5", "claude-haiku-4-5"],
	"fallbackModel": ["claude-haiku-4-5"]
}
```

`fallbackModel` names the models to try when the primary one is unavailable, capped at three.

`availableModels` only filters the rows Claude Code already has — it never creates new ones. To put a non-Claude model in the picker, see [Adding Non-Claude Models to the Picker](#adding-non-claude-models-to-the-picker).

## Fetching Models From LLM Gateway [#fetching-models-from-llm-gateway]

Claude Code can populate its `/model` picker directly from our catalog instead of you hardcoding IDs. Set the discovery flag alongside the base URL:

```bash
export ANTHROPIC_BASE_URL=https://api.llmgateway.io
export ANTHROPIC_AUTH_TOKEN=llmgtwy_your_api_key_here
export CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1
```

On startup Claude Code calls our [`/v1/models`](https://api.llmgateway.io/v1/models) endpoint and adds what it returns to the picker, labeled **From gateway**. Entries are cached in `~/.claude/cache/gateway-models.json` and refreshed on each launch, so a failed lookup falls back to the previous list rather than breaking your session. Requires Claude Code v2.1.129 or later.

The [LLM Gateway CLI](https://docs.llmgateway.io/developers/cli#configure) can apply this whole setup (base URL, auth token, and the discovery flag) to your `~/.claude/settings.json` in one command:

```bash
npx @llmgateway/cli configure claude
```

<Callout type="warn">
  Claude Code drops discovered models whose ID does not start with `claude` or
  `anthropic`, before they ever reach the picker — the cache it writes to
  `~/.claude/cache/gateway-models.json` contains only the surviving entries.
  Discovery therefore surfaces just the Claude models in our catalog. GPT-5,
  Gemini, and custom models are filtered out by the client, not by the gateway.
</Callout>

### Adding Non-Claude Models to the Picker [#adding-non-claude-models-to-the-picker]

The `/model&#x60; picker is a Claude-model list. Its own header says so: &#x2A;"Switch between Claude models… For other/previous model names, specify with `--model`."* Neither gateway discovery nor `availableModels` adds a non-Claude row — `availableModels` filters the rows Claude Code already has rather than creating new ones.

`ANTHROPIC_CUSTOM_MODEL_OPTION` is the one setting that adds a non-Claude row:

```bash
export ANTHROPIC_CUSTOM_MODEL_OPTION=gemini-3.5-flash
export ANTHROPIC_CUSTOM_MODEL_OPTION_NAME="Gemini 3.5 Flash"
export ANTHROPIC_CUSTOM_MODEL_OPTION_DESCRIPTION="Routed through LLM Gateway"
```

The entry appears at the bottom of the picker under your chosen name. Only one is supported at a time, so it suits a single alternate model rather than a menu. If you also set `availableModels`, include this ID there or it will be filtered back out.

### Selecting Any Other Model [#selecting-any-other-model]

For everything else — including [custom models](https://docs.llmgateway.io/features/custom-providers) — name the model directly. These paths accept any ID our gateway routes, with no picker row involved:

```bash
export ANTHROPIC_MODEL=gemini-3.5-flash   # session default
claude --model gemini-3.5-flash           # single session
```

To rotate between several non-Claude models, keep a per-project `.claude/settings.json` with the `model` you want for that repo.

## Environment Variables [#environment-variables]

### ANTHROPIC\_MODEL [#anthropic_model]

Specifies the main model to use for primary requests.

```bash
export ANTHROPIC_MODEL=gpt-5
```

### Complete Configuration Example [#complete-configuration-example]

```bash
export ANTHROPIC_BASE_URL=https://api.llmgateway.io
export ANTHROPIC_AUTH_TOKEN=llmgtwy_your_api_key_here
export ANTHROPIC_MODEL=gpt-5
export ANTHROPIC_SMALL_FAST_MODEL=gpt-5-nano
```

## Making Manual API Requests [#making-manual-api-requests]

If you want to test the endpoint directly, you can make manual requests:

```bash
curl -X POST "https://api.llmgateway.io/v1/messages" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "max_tokens": 100
  }'
```

### Response Format [#response-format]

The endpoint returns responses in Anthropic's message format:

```json
{
	"id": "msg_abc123",
	"type": "message",
	"role": "assistant",
	"model": "gpt-5",
	"content": [
		{
			"type": "text",
			"text": "Hello! I'm doing well, thank you for asking. How can I help you today?"
		}
	],
	"stop_reason": "end_turn",
	"stop_sequence": null,
	"usage": {
		"input_tokens": 13,
		"output_tokens": 20
	}
}
```

## What You Get [#what-you-get]

* **Any model in Claude Code** — GPT-5 for heavy lifting, GPT-4o Mini for routine tasks
* **Cost visibility** — See exactly what each coding agent costs
* **One bill** — Stop managing separate accounts for OpenAI, Anthropic, Google
* **Response caching** — Repeated requests (like linting the same file) hit cache
* **Discounts** — Check [discounted models](https://llmgateway.io/models?discounted=true) for savings up to 90%

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Cline Integration
URL: https://llmgateway.io/guides/cline


[Cline](https://cline.bot) is an autonomous AI coding assistant that lives in your VS Code editor. It can create and edit files, run terminal commands, and help you build complex projects. You can configure Cline to use LLM Gateway for access to multiple AI providers with unified billing and cost tracking.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* VS Code based IDE installed
* An LLM Gateway API key

## Setup [#setup]

Cline supports OpenAI-compatible API endpoints, making it straightforward to integrate with LLM Gateway.

<Steps>
  <Step>
    ### Install Cline Extension [#install-cline-extension]

    1. Open VS Code
    2. Go to the Extensions view (Cmd/Ctrl + Shift + X)
    3. Search for "Cline"
    4. Click **Install** on the Cline extension

        <img alt="Install Cline Extension" src="__img0" />
  </Step>

  <Step>
    ### Open Cline Settings [#open-cline-settings]

    1. Click on the Cline icon in the VS Code sidebar
    2. Click the settings gear icon in the Cline panel

        <img alt="Cline Settings" src="__img1" />
  </Step>

  <Step>
    ### Configure API Provider [#configure-api-provider]

    1. In the API Provider dropdown, select **OpenAI Compatible**
    2. Enter the following details:
       * **Base URL**: `https://api.llmgateway.io/v1`
       * **API Key**: Your LLM Gateway API key
       * **Model ID**: Choose a model (e.g., `claude-opus-4-5-20251101`, `gpt-5.2`, `gemini-3-pro-preview`, `deepseek-3.2`). See [provider-specific routing](https://docs.llmgateway.io/features/routing#provider-specific-routing) for more options.

        <img alt="Configure API Provider" src="__img2" />
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    1. Open a project in VS Code
    2. Click on the Cline icon in the sidebar
    3. Type a message like "Create a hello world function in Python"
    4. Cline should respond and offer to create the file

        <img alt="Test Cline" src="__img3" />

    All requests will now be routed through LLM Gateway.
  </Step>
</Steps>

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

## Features [#features]

Once configured, you can use all of Cline's features with LLM Gateway:

### Autonomous Coding [#autonomous-coding]

* Create new files and projects from scratch
* Edit existing code based on natural language instructions
* Refactor and improve code quality

### Terminal Commands [#terminal-commands]

* Run build commands, tests, and scripts
* Install dependencies
* Execute any terminal operation

### File Management [#file-management]

* Create, read, and modify files
* Navigate your codebase
* Search for relevant code

## Model Selection Tips [#model-selection-tips]

### Using Provider-Specific Models [#using-provider-specific-models]

To use a specific provider's version of a model, prefix the model ID with the provider name. See [provider-specific routing](https://docs.llmgateway.io/features/routing#provider-specific-routing) for more options.

### Using Discounted Models [#using-discounted-models]

LLM Gateway offers discounted access to some models. Find them on the [models page](https://llmgateway.io/models?view=grid\&filters=1\&discounted=true) and copy the model ID.

### Using Free Models [#using-free-models]

Some models are available for free. Browse them on the [models page](https://llmgateway.io/models?view=grid\&filters=1\&free=true).

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>

## Benefits of Using LLM Gateway with Cline [#benefits-of-using-llm-gateway-with-cline]

* **Multi-Provider Access**: Use models from OpenAI, Anthropic, Google, and more through a single API
* **Cost Control**: Track and limit your AI spending with detailed usage analytics
* **Unified Billing**: One account for all providers instead of managing multiple API keys
* **Caching**: Reduce costs with response caching for repeated requests
* **Analytics**: Monitor usage patterns and costs in the dashboard


# Codex CLI Integration
URL: https://llmgateway.io/guides/codex-cli


Codex CLI is OpenAI's open-source terminal coding agent. By default it connects to OpenAI's API, but with LLM Gateway you can route it through a single gateway—use GPT-5.3 Codex, Gemini, Claude, or any of 200+ models while keeping full cost visibility.

One config file. No code changes. Full cost tracking in your dashboard.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for LLM Gateway [#sign-up-for-llm-gateway]

    [Sign up free](https://llmgateway.io/signup) — no credit card required. Create an API key in the dashboard and copy it when shown.
  </Step>

  <Step>
    ### Log Out of ChatGPT [#log-out-of-chatgpt]

    If you're logged into ChatGPT in Codex CLI, the stored session will override your custom config. Log out first:

    ```bash
    codex logout
    ```
  </Step>

  <Step>
    ### Create Config File [#create-config-file]

    Create or edit `~/.codex/config.toml`:

    ```bash
    model = "auto"
    model_reasoning_effort = "high"
    openai_base_url = "https://api.llmgateway.io/v1"
    ```
  </Step>

  <Step>
    ### Run Codex CLI [#run-codex-cli]

    ```bash
    codex
    ```

    On first launch, Codex will prompt you for authentication. Select **Provide your own API key**, then enter your LLM Gateway API key (starts with `llmgtwy_`).

    All requests will now be routed through LLM Gateway.
  </Step>
</Steps>

## Why This Works [#why-this-works]

LLM Gateway's `/v1` endpoint is fully OpenAI-compatible. Codex CLI sends requests to our gateway instead of OpenAI directly, and we route them to the right provider behind the scenes. This means:

* **Use any model** — GPT-5.3 Codex, Gemini, Claude, or 180+ others
* **Keep your workflow** — Codex CLI doesn't know the difference
* **Track costs** — Every request appears in your LLM Gateway dashboard
* **Automatic caching** — Repeated requests hit cache, saving money

## Configuration Explained [#configuration-explained]

### Base URL [#base-url]

The `openai_base_url` field points Codex CLI to LLM Gateway instead of OpenAI:

```bash
openai_base_url = "https://api.llmgateway.io/v1"
```

### Model Selection [#model-selection]

Use `auto` to let LLM Gateway pick the best model, or set a specific one from the [models page](https://llmgateway.io/models):

```bash
model = "auto"
# or pick a specific model
model = "gpt-5.3-codex"
```

### Reasoning Effort [#reasoning-effort]

Control how much reasoning the model uses. Options are `low`, `medium`, and `high`:

```bash
model_reasoning_effort = "high"
```

## Choosing Models [#choosing-models]

Use `auto` to let LLM Gateway pick the best model automatically, or choose a specific one from the [models page](https://llmgateway.io/models):

```bash
# let LLM Gateway pick the best model
model = "auto"

# or pick a specific model
model = "gpt-5.3-codex"
```

## What You Get [#what-you-get]

* **Any model in Codex CLI** — GPT-5.3 Codex for heavy lifting, lighter models for routine tasks
* **Cost visibility** — See exactly what each coding agent costs
* **One bill** — Stop managing separate accounts for OpenAI, Anthropic, Google
* **Response caching** — Repeated requests hit cache automatically
* **Discounts** — Check [discounted models](https://llmgateway.io/models?discounted=true) for savings up to 90%

## Troubleshooting [#troubleshooting]

### Data retention required [#data-retention-required]

Older versions of LLM Gateway rejected Responses API requests with:

```
The Responses API requires data retention to be enabled.
```

This is no longer the case — the Responses API (which Codex CLI uses) works regardless of your organization's data retention policy. If you still see this error, the gateway you are talking to is running an outdated version; on a self-hosted deployment, update to the latest release.

### Authentication errors [#authentication-errors]

If you see `401 Unauthorized` or requests going to `api.openai.com` instead of LLM Gateway:

1. Make sure you've run `codex logout` to clear any ChatGPT session
2. Verify `openai_base_url` is set in `~/.codex/config.toml`
3. When Codex prompts for authentication, select **Provide your own API key** and enter your LLM Gateway key (starts with `llmgtwy_`)

### Model not found [#model-not-found]

Verify the model ID matches exactly what's listed on the [models page](https://llmgateway.io/models). Model IDs are case-sensitive.

### Connection issues [#connection-issues]

Check that `openai_base_url` is set to `https://api.llmgateway.io/v1` (note the `/v1` at the end).

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Continue CLI Integration
URL: https://llmgateway.io/guides/continue


[Continue](https://docs.continue.dev) is an open-source AI code assistant available as a CLI tool. By configuring it to use LLM Gateway, you get access to 200+ models from 40+ providers with unified cost tracking.

One config file. Any model. Full cost visibility.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* An LLM Gateway API key — [sign up free](https://llmgateway.io/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install Continue CLI [#install-continue-cli]

    Install Continue CLI globally:

    ```bash
    npm install -g @continuedev/cli
    ```

        <img alt="Installing Continue CLI" src="__img0" />
  </Step>

  <Step>
    ### Get Your API Key [#get-your-api-key]

    [Sign up](https://llmgateway.io/signup) or log in to your LLM Gateway dashboard. Navigate to **API Keys** and create a new key. Copy it — it starts with `llmgtwy_`.
  </Step>

  <Step>
    ### Create a Config File [#create-a-config-file]

    Create the Continue config directory and config file:

    ```bash
    mkdir -p ~/.continue
    ```

    Then create `~/.continue/config.yaml` with your LLM Gateway configuration:

    ```yaml
    name: llmgateway
    version: 0.0.1
    models:
      - name: claude-sonnet-4-6
        provider: openai
        model: claude-sonnet-4-6
        apiBase: https://api.llmgateway.io/v1
        apiKey: llmgtwy_your-api-key-here
    ```

        <img alt="Editing config.yaml" src="__img1" />

    <Callout type="info">
      Replace `llmgtwy_your-api-key-here` with your actual API key from the
      dashboard.
    </Callout>
  </Step>

  <Step>
    ### Add More Models (Optional) [#add-more-models-optional]

    Add as many models as you want from the [models page](https://llmgateway.io/models):

    ```yaml
    name: llmgateway
    version: 0.0.1
    models:
      - name: claude-sonnet-4-6
        provider: openai
        model: claude-sonnet-4-6
        apiBase: https://api.llmgateway.io/v1
        apiKey: llmgtwy_your-api-key-here
      - name: gpt-5.5
        provider: openai
        model: gpt-5.5
        apiBase: https://api.llmgateway.io/v1
        apiKey: llmgtwy_your-api-key-here
      - name: gemini-3.1-pro
        provider: openai
        model: gemini-3.1-pro
        apiBase: https://api.llmgateway.io/v1
        apiKey: llmgtwy_your-api-key-here
    ```

    All models use `provider: openai` since LLM Gateway exposes an OpenAI-compatible API.
  </Step>

  <Step>
    ### Start Using Continue [#start-using-continue]

    Launch Continue CLI with the `--config` flag pointing to your config file:

    ```bash
    cn --config ~/.continue/config.yaml
    ```

        <img alt="Continue CLI running with LLM Gateway" src="__img2" />

    All requests now route through LLM Gateway. You'll see usage, costs, and logs in your dashboard.
  </Step>
</Steps>

## Why Use LLM Gateway with Continue [#why-use-llm-gateway-with-continue]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://llmgateway.io/models?discounted=true) for savings up to 90%

## Configuration Details [#configuration-details]

### Provider Setting [#provider-setting]

Always use `provider: openai` in your Continue config. LLM Gateway exposes an OpenAI-compatible API, so Continue's OpenAI provider handles all models correctly — including Claude, Gemini, and others.

### Project-Specific Config [#project-specific-config]

Place a `.continue/config.yaml` in your project root to override the global config for that project:

```yaml
name: project-config
version: 0.0.1
models:
  - name: gpt-5.5
    provider: openai
    model: gpt-5.5
    apiBase: https://api.llmgateway.io/v1
    apiKey: llmgtwy_your-api-key-here
```

### Using with the --config Flag [#using-with-the---config-flag]

Point to any config file:

```bash
cn --config path/to/config.yaml
```

## Switching Models [#switching-models]

Add multiple models to your config and switch between them in the Continue interface. In the CLI, you can specify a model with the `--model` flag if supported, or update your config file.

## Locking to a Specific Provider [#locking-to-a-specific-provider]

By default, LLM Gateway automatically fails over to alternative providers if your chosen provider is experiencing downtime. To disable fallback, add a custom header:

```yaml
models:
  - name: claude-sonnet-4-6
    provider: openai
    model: claude-sonnet-4-6
    apiBase: https://api.llmgateway.io/v1
    apiKey: llmgtwy_your-api-key-here
    requestOptions:
      headers:
        X-No-Fallback: "true"
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.llmgateway.io/features/routing) for details.
</Callout>

## Troubleshooting [#troubleshooting]

### "Failed to parse config" error [#failed-to-parse-config-error]

Make sure your config file includes `name` and `version` fields at the top level:

```yaml
name: llmgateway
version: 0.0.1
models:
  - ...
```

### Onboarding wizard still appears [#onboarding-wizard-still-appears]

If running `cn` without `--config` shows an onboarding prompt, create the sentinel file to skip it:

```bash
touch ~/.continue/.onboarding_complete
```

Or always launch with the `--config` flag to bypass onboarding entirely.

### Model not found [#model-not-found]

Verify the model ID matches exactly what's listed on the [models page](https://llmgateway.io/models). Model IDs are case-sensitive.

### Connection timeout [#connection-timeout]

Check that `apiBase` is set to `https://api.llmgateway.io/v1` (note the `/v1` at the end).

### Authentication errors [#authentication-errors]

Make sure your `apiKey` starts with `llmgtwy_` and is valid. Check your [dashboard](https://llmgateway.io/dashboard) to confirm the key is active.

### Provider must be "openai" [#provider-must-be-openai]

LLM Gateway uses an OpenAI-compatible API. Even when using Claude or Gemini models, set `provider: openai` in your Continue config. The gateway handles routing to the correct upstream provider.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Crush Integration
URL: https://llmgateway.io/guides/crush


[Crush](https://github.com/charmbracelet/crush) is Charm's glamorous open-source AI coding agent for your terminal. Connecting it to LLM Gateway takes a single provider entry in Crush's config — and gives you access to 200+ models from 40+ providers through one API key, with every request tracked in your dashboard.

## Prerequisites [#prerequisites]

* Crush installed:

  ```bash
  # Homebrew
  brew install charmbracelet/tap/crush

  # NPM
  npm install -g @charmland/crush
  ```

  See the [Crush README](https://github.com/charmbracelet/crush#installation) for Windows, Arch, Nix, and FreeBSD instructions.

* An LLM Gateway API key — [sign up](https://llmgateway.io/signup) and create one from your dashboard

## Setup [#setup]

<Steps>
  <Step>
    ### Set Your API Key [#set-your-api-key]

    Export your LLM Gateway API key in your shell:

    ```bash
    export LLMGATEWAY_API_KEY=your-api-key-here
    ```

    Add it to your shell profile (`~/.zshrc` or `~/.bashrc`) to make it permanent.
  </Step>

  <Step>
    ### Add LLM Gateway as a Provider [#add-llm-gateway-as-a-provider]

    Create a `crush.json` in your project directory (or `~/.config/crush/crush.json` for a global setup):

    ```json
    {
    	"$schema": "https://charm.land/crush.json",
    	"providers": {
    		"llmgateway": {
    			"name": "LLM Gateway",
    			"type": "openai-compat",
    			"base_url": "https://api.llmgateway.io/v1",
    			"api_key": "$LLMGATEWAY_API_KEY"
    		}
    	}
    }
    ```

    Crush auto-discovers all available models from LLM Gateway's `/v1/models` endpoint, so there is no model list to maintain.
  </Step>

  <Step>
    ### Start Coding [#start-coding]

    Launch Crush:

    ```bash
    crush
    ```

    Select "LLM Gateway" from the provider list, pick a model, and start building. You can switch models at any time from within Crush.
  </Step>
</Steps>

## Why Use LLM Gateway with Crush [#why-use-llm-gateway-with-crush]

* **200+ models** — GPT-5, Claude, Gemini, Kimi, GLM, and more from 40+ providers
* **One API key** — Stop juggling credentials for every provider
* **Cost tracking** — See what each coding session costs in your dashboard
* **Automatic fallback** — Requests route to a healthy provider if one is down
* **Volume discounts** — The more you use, the more you save

## Locking to a Specific Provider [#locking-to-a-specific-provider]

By default, LLM Gateway automatically fails over to alternative providers if your chosen provider is experiencing downtime. If you want to lock into a specific provider/model mapping — for example to guarantee a fixed price or to always use a single provider — add the `X-No-Fallback` header to your provider entry:

```json
{
	"$schema": "https://charm.land/crush.json",
	"providers": {
		"llmgateway": {
			"name": "LLM Gateway",
			"type": "openai-compat",
			"base_url": "https://api.llmgateway.io/v1",
			"api_key": "$LLMGATEWAY_API_KEY",
			"extra_headers": {
				"X-No-Fallback": "true"
			}
		}
	}
}
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.llmgateway.io/features/routing) for details.
</Callout>

## Switching Models [#switching-models]

Use Crush's model picker to switch between any of the discovered models.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

## Troubleshooting [#troubleshooting]

### 401 Unauthorized [#401-unauthorized]

Verify `LLMGATEWAY_API_KEY` is exported in the shell where you launch Crush and that the key is active in your [dashboard](https://llmgateway.io/dashboard).

### 404 Not Found [#404-not-found]

Verify your `base_url` is set to `https://api.llmgateway.io/v1` (note the `/v1` at the end).

### Models not showing up [#models-not-showing-up]

Model discovery runs when Crush starts — restart Crush after adding or changing the provider config.

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Cursor Integration
URL: https://llmgateway.io/guides/cursor


Cursor is an AI-powered code editor built on VSCode. You can point Cursor's custom OpenAI base URL at LLM Gateway to use any of our 200+ models in the AI panel — both **plan mode** and **agent mode**.

<Callout type="info">
  **Plan and agent mode.** Cursor routes its AI panel — plan mode and agent mode
  — through the custom API key + base URL, so both work with LLM Gateway. Tab
  autocomplete and inline edit (Cmd/Ctrl + K) remain locked to Cursor's own
  backend and will not route through the gateway. If you want every request in
  your workflow to go through LLM Gateway, use a terminal agent like [Claude
  Code](https://docs.llmgateway.io/guides/claude-code), [Codex CLI](https://docs.llmgateway.io/guides/codex-cli),
  [Cline](https://docs.llmgateway.io/guides/cline), [Continue CLI](https://docs.llmgateway.io/guides/continue), or [Hermes
  Agent](https://docs.llmgateway.io/guides/hermes-agent).
</Callout>

<img alt="Cursor with LLM Gateway" src="__img0" />

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* An LLM Gateway account with an API key
* Cursor IDE installed
* Basic understanding of Cursor's AI features

## Setup [#setup]

Cursor supports OpenAI-compatible API endpoints, making it easy to integrate with LLM Gateway.

<Steps>
  <Step>
    ### Get Your API Key [#get-your-api-key]

    1. Log in to your [LLM Gateway dashboard](https://llmgateway.io/dashboard)
    2. Navigate to **API Keys** section
    3. Create a new API key and copy the key

        <img alt="LLM Gateway API Keys" src="__img1" />
  </Step>

  <Step>
    ### Configure Cursor Settings [#configure-cursor-settings]

    1. Open Cursor and go to **Settings** then Click on "Cursor Settings"
    2. Click on "Models"
    3. Click on "Add OpenAI API Key"

        <img alt="Cursor Settings" src="__img2" />

    3. Scroll down to **OpenAI API Key** section
    4. Click on **Add OpenAI API Key**

        <img alt="Cursor API Key Input" src="__img3" />

    5. Enter your LLM Gateway API key

    6. In the same Models settings, find the **Override OpenAI Base URL** option

    7. Enable the override option

    8. Enter the LLM Gateway endpoint: `https://api.llmgateway.io/v1`
  </Step>

  <Step>
    ### Select Models [#select-models]

    1. In the **Models** section, you can now select from available models
    2. Choose any [LLM Gateway supported model](https://llmgateway.io/models):

        <img alt="Cursor Model Selection" src="__img4" />

    * For chat: Use models like `gpt-5`, `gpt-4o`, `claude-sonnet-4-5`
    * For custom models: Add the provider name before the model name (e.g. `custom/my-model`)
    * For discounted models: copy the ids from from the [models page](https://llmgateway.io/models?view=grid\&filters=1\&discounted=true)
    * For free models: copy the ids from from the [models page](https://llmgateway.io/models?view=grid\&filters=1\&free=true)
    * For reasoning models: copy the ids from from the [models page](https://llmgateway.io/models?view=grid\&filters=1\&reasoning=true)
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    1. Open any code file in Cursor
    2. Try using the AI chat (Cmd/Ctrl + L) in plan mode
    3. Then switch the panel to agent mode and run a small multi-file task

        <img alt="Cursor AI Chat" src="__img5" />
        <img alt="Cursor AI Chat 2" src="__img6" />

    All AI requests will now be routed through LLM Gateway.
  </Step>
</Steps>

## What Works (and What Doesn't) [#what-works-and-what-doesnt]

Cursor honors the custom OpenAI base URL for the **AI panel** (Cmd/Ctrl + L) — both plan mode and agent mode. Tab autocomplete and inline edit still use Cursor's own backend, even after you save the LLM Gateway key.

### Works through LLM Gateway [#works-through-llm-gateway]

* **Plan mode (Cmd/Ctrl + L)** — Ask questions, plan changes, get explanations, debug.
* **Agent mode** — Cursor's coding agent runs multi-file edits with your gateway models.

All of these requests route through LLM Gateway and appear in your dashboard.

### Does NOT work through LLM Gateway [#does-not-work-through-llm-gateway]

* **Inline Edit (Cmd/Ctrl + K)** — Locked to Cursor's backend.
* **Autocomplete / Tab completion** — Locked to Cursor's backend.

If you want every request in your workflow — including edits and completions — to route through LLM Gateway, use [Claude Code](https://docs.llmgateway.io/guides/claude-code), [Codex CLI](https://docs.llmgateway.io/guides/codex-cli), [Cline](https://docs.llmgateway.io/guides/cline), [Continue CLI](https://docs.llmgateway.io/guides/continue), or [Hermes Agent](https://docs.llmgateway.io/guides/hermes-agent).

### Model Routing [#model-routing]

With LLM Gateway's [routing features](https://docs.llmgateway.io/features/routing), you can:

* **Chooses cost-effective models** by default for optimal price-to-performance ratio
* **Automatically scales to more powerful models** based on your request's context size
* **Handles large contexts intelligently** by selecting models with appropriate context windows

## Troubleshooting [#troubleshooting]

### Authentication Errors [#authentication-errors]

If you see authentication errors:

* Verify your API key is correct
* Check that the base URL is set to `https://api.llmgateway.io/v1`
* Ensure your LLM Gateway account has sufficient credits

### Model Not Found [#model-not-found]

If you see "model not found" errors:

* Verify the model ID exists in the [models page](https://llmgateway.io/models)
* Check that you're using the correct model name format
* Some models may require specific provider configurations in your LLM Gateway dashboard

### Slow Responses [#slow-responses]

If responses are slow:

* Check your internet connection
* Monitor your usage in the LLM Gateway dashboard
* Switch to a faster chat model from the [models page](https://llmgateway.io/models)

### Autocomplete / inline edit still uses Cursor's models [#autocomplete--inline-edit-still-uses-cursors-models]

This is expected. Cursor routes the AI panel — plan mode and agent mode — through the custom API key, but tab autocomplete and inline edit are locked to Cursor's own backend. See [What Works (and What Doesn't)](#what-works-and-what-doesnt) above.

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>

## Benefits of Using LLM Gateway with Cursor [#benefits-of-using-llm-gateway-with-cursor]

* **Multi-Provider Access**: Use models from OpenAI, Anthropic, Google, Open-source models and more
* **Cost Control**: Track and limit your AI spending with detailed usage analytics
* **Caching**: Reduce costs with response caching
* **Analytics**: Monitor usage patterns and costs


# DevPass Code
URL: https://llmgateway.io/guides/devpass-code


[DevPass Code](https://github.com/theopenco/devpass-code) is a terminal coding agent that talks **only** to LLM Gateway. It's a rebranded fork of [opencode](https://github.com/anomalyco/opencode) (MIT), trimmed down to two providers that both point at LLM Gateway's OpenAI-compatible API. Every request is tagged with `x-source: devpass-code`, so DevPass usage is attributed correctly and you get access to all \~190 text models LLM Gateway offers.

Log in once from your terminal with a single click in the browser — no copy-pasting API keys unless you want to.

<Callout type="info">
  **On a DevPass plan?** Use canonical model IDs without a provider prefix
  (`claude-sonnet-4-5`, not `anthropic/claude-sonnet-4-5`) — provider-pinned
  routing is not available on coding plans; the gateway picks the provider for
  you.
</Callout>

## Install [#install]

Install DevPass Code globally from npm:

```bash
npm i -g devpass-code
```

Or with Homebrew:

```bash
brew install theopenco/tap/devpass-code
```

Other options: an install script (`curl -fsSL https://raw.githubusercontent.com/theopenco/devpass-code/main/install | bash`), an AUR package (`devpass-code-bin`), a Docker image (`ghcr.io/theopenco/devpass-code`), and Windows binaries on [GitHub releases](https://github.com/theopenco/devpass-code/releases).

To run from source instead, clone the repository, `bun install`, then:

```bash
bun run packages/devpass-code/src/index.ts --help
```

## Authenticate [#authenticate]

DevPass Code supports one-click browser login, just like Claude Code. Start the login flow:

```bash
devpass-code auth login
```

<Steps>
  <Step>
    ### Pick a provider [#pick-a-provider]

    Choose either **LLM Gateway** (pay-as-you-go with your own API key) or **LLM Gateway DevPass** (the DevPass coding subscription). See [Providers](#providers) below for the difference.
  </Step>

  <Step>
    ### Log in with browser [#log-in-with-browser]

    Select &#x2A;*"Log in with browser."** DevPass Code opens [https://llmgateway.io/connect/cli](https://llmgateway.io/connect/cli) and starts a local loopback server.
  </Step>

  <Step>
    ### Approve in the browser [#approve-in-the-browser]

    Approve the request in your browser. The API key is delivered back to the local loopback server automatically — nothing to copy or paste.
  </Step>

  <Step>
    ### Start coding [#start-coding]

    Your credentials are saved to `~/.local/share/devpass-code/auth.json`. You're ready to go.
  </Step>
</Steps>

<Callout type="info">
  Prefer to paste a key? Choose &#x2A;*"Paste an API key"** during `devpass-code auth
  	login` and enter a key from your [LLM Gateway
  dashboard](https://llmgateway.io/dashboard). You can also set the
  `LLMGATEWAY_API_KEY` environment variable to skip the prompt entirely.
</Callout>

## Providers [#providers]

DevPass Code ships exactly two providers. Both hit `https://api.llmgateway.io/v1` (OpenAI-compatible) and expose the same \~190 text models — the only difference is how usage is billed.

| Provider                | Billing                                                                                                 |
| ----------------------- | ------------------------------------------------------------------------------------------------------- |
| **LLM Gateway**         | Pay-as-you-go with your own API key                                                                     |
| **LLM Gateway DevPass** | The DevPass coding subscription — billing is handled automatically by the gateway based on your account |

<Callout type="info">
  Both providers route through the same gateway, so you can switch between them
  at any time without changing anything else. View all available models on the
  [models page](https://llmgateway.io/models).
</Callout>

## Use it [#use-it]

Once authenticated, start DevPass Code in any project directory and describe what you want to build. All requests route through LLM Gateway with `x-source: devpass-code`, so every prompt and its cost appear in your [dashboard](https://llmgateway.io/dashboard).

## Configuration [#configuration]

* **Config file** — Place a `devpass-code.json` in your project (or global config directory) to customize models and behavior.
* **`LLMGATEWAY_API_KEY`** — Provide your LLM Gateway API key without running the login flow.
* **`DEVPASS_APP_URL`** — Override the app URL used for browser login (defaults to `https://llmgateway.io`).

## Why Use LLM Gateway with DevPass Code [#why-use-llm-gateway-with-devpass-code]

* **\~190 text models** — GPT-5, Claude, Gemini, Llama, and more, all through one endpoint
* **One-click login** — Approve in the browser, key delivered automatically
* **DevPass billing** — The DevPass subscription is handled automatically by the gateway
* **Cost tracking** — See exactly what each coding session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically

<Callout type="info">
  Source, issues, and releases live at
  [github.com/theopenco/devpass-code](https://github.com/theopenco/devpass-code).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Empryo Integration
URL: https://llmgateway.io/guides/empryo


[Empryo](https://empryo.com) is a graph-powered AI coding agent — it parses your codebase into a code-intelligence graph and edits symbols, not strings. It runs as a terminal UI, a desktop app, and a headless CLI on macOS, Linux, and Windows, and LLM Gateway is built in: one browser login (or one pasted key) connects all \~190 text models on the gateway. Every request is tagged with `x-source: empryo`, so usage is attributed correctly in your dashboard and on your DevPass plan.

<Callout type="info">
  **On a DevPass plan?** Log in with `/login llmgateway-devpass` and use
  canonical model IDs (`claude-opus-5`, not `anthropic/claude-opus-5`) —
  provider-pinned routing is not available on coding plans; the gateway picks
  the provider for you.
</Callout>

## Install [#install]

macOS / Linux:

```bash
curl -fsSL https://empryo.com/install.sh | bash
```

Windows (PowerShell):

```powershell
irm https://empryo.com/install.ps1 | iex
```

Each script downloads the release's `SHA256SUMS.txt`, checks its RSA-4096 signature against a key pinned in the script, and verifies the binary's checksum before installing it — the one-liner itself is a bootstrapper, so if you'd rather not pipe it into your shell, download it first and read it, or grab a binary and its checksum from the [download page](https://empryo.com/download) and verify by hand.

## Setup [#setup]

<Steps>
  <Step>
    ### Launch Empryo [#launch-empryo]

    Start Empryo in any project directory:

    ```bash
    empryo
    ```
  </Step>

  <Step>
    ### Log in with your browser [#log-in-with-your-browser]

    Type `/login`. Empryo opens [llmgateway.io/connect/cli](https://llmgateway.io/connect/cli) and starts a local loopback server — approve the connection in the browser and the minted API key is delivered straight back to Empryo. Nothing to copy or paste.

    On a DevPass coding subscription, run `/login llmgateway-devpass` instead — same flow, billed to your plan.
  </Step>

  <Step>
    ### Pick a model [#pick-a-model]

    Open the model picker with `/models`. LLM Gateway's models are grouped by upstream provider, with live pricing, context windows, and output limits pulled from the gateway's own catalog — what you see is what you're billed.
  </Step>

  <Step>
    ### Start coding [#start-coding]

    You're connected. Every prompt and its cost appears in your [dashboard](https://llmgateway.io/dashboard), attributed to Empryo.
  </Step>
</Steps>

<Callout type="info">
  Prefer to paste a key? Use `/keys` inside the TUI, run `empryo --set-key
  	llmgateway <key>` from a shell, or set the `LLM_GATEWAY_API_KEY`
  environment variable. If both a browser login
  and a pasted key exist, Empryo prefers the login — set `llmgatewayAuth:
  	"key"` in your config to flip that.
</Callout>

## One connection, three surfaces [#one-connection-three-surfaces]

The gateway connection is shared by everything Empryo ships:

* **Terminal UI** — `/login`, `/keys`, and `/models` as above.
* **Desktop app** — the same browser login (from onboarding or the keys drawer) and the same grouped model picker.
* **Headless CLI** — for scripts and CI:

```bash
empryo --headless "fix the failing test" --model llmgateway/claude-opus-5
```

Add `--json` for a structured result or `--events` for a real-time JSONL stream.

## Model routing [#model-routing]

Empryo passes model IDs to the gateway verbatim, so both forms work:

* **Canonical** (`claude-opus-5`) — the gateway routes to the best available provider, with automatic failover. Required on DevPass plans.
* **Provider-pinned** (`anthropic/claude-opus-5`) — pay-as-you-go keys can pin a specific upstream deployment. See the [routing docs](https://docs.llmgateway.io/features/routing#provider-specific-routing) for the full syntax.

Claude models take the gateway's Anthropic-native endpoint (`/v1/messages`) rather than the OpenAI-compatible one, so extended thinking and prompt caching (`cache_control`) survive the hop. Other models ride `/v1/chat/completions`, with reasoning effort injected where the model supports it.

## DevPass meters, natively [#devpass-meters-natively]

With a DevPass key connected, Empryo reads the gateway's `GET /v1/key` endpoint and renders your plan meters in the product: monthly credits and the weekly premium fair-use window show up as a footer badge in the TUI and in the desktop usage drawer, next to Empryo's own usage ledger (`/usage`), which tracks what every session cost locally.

## Why use LLM Gateway with Empryo [#why-use-llm-gateway-with-empryo]

* **\~190 text models** — Claude, GPT, Gemini, Grok, DeepSeek, and more through one login
* **Live pricing in the picker** — rates, context, and output caps come from the gateway's catalog, not a stale copy
* **Native Anthropic wire for Claude** — thinking and prompt caching intact
* **Attribution built in** — every request carries `x-source: empryo`, so the dashboard shows exactly what the agent spent
* **DevPass aware** — plan meters render inside Empryo, and billing is handled by the gateway

## Troubleshooting [#troubleshooting]

### "LLM Gateway is set to use the browser login, but you're not connected" [#llm-gateway-is-set-to-use-the-browser-login-but-youre-not-connected]

Run `/login`, or switch to a pasted key with `/keys` (and set `llmgatewayAuth: "key"` if you want the key to win permanently).

### "Direct provider routing is not available on coding plans" [#direct-provider-routing-is-not-available-on-coding-plans]

Your key belongs to a DevPass plan and you picked a provider-pinned ID. Choose the canonical form of the model instead — `claude-opus-5`, not `anthropic/claude-opus-5`.

### Models show no price [#models-show-no-price]

The catalog is fetched live from the gateway at startup. Check your connection and that your key is valid in the [dashboard](https://llmgateway.io/dashboard), then restart Empryo.

<Callout type="info">
  Empryo's own documentation lives at
  [empryo.com/docs](https://empryo.com/docs). Need help on the gateway side?
  Join our [Discord community](https://llmgateway.io/discord).
</Callout>


# GitHub Copilot App Integration
URL: https://llmgateway.io/guides/github-copilot


The [GitHub Copilot app](https://github.com/features/ai/github-app) is GitHub's desktop app for agent-driven development — start agent sessions from issues, pull requests, or prompts, run parallel workflows in isolated workspaces, and merge PRs without leaving the app. It supports bring your own key (BYOK), so you can run agent sessions against your own model provider.

Add LLM Gateway as that provider and every session can use Claude, Gemini, GPT, or any model in the [catalog](https://llmgateway.io/models) that supports tool calling — with full cost visibility in your dashboard.

One provider entry. No config files. Works on any Copilot plan, or with no Copilot plan at all.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for LLM Gateway [#sign-up-for-llm-gateway]

    [Sign up free](https://llmgateway.io/signup) — no credit card required. Create an API key in the dashboard and copy it when shown (it starts with `llmgtwy_`).
  </Step>

  <Step>
    ### Install the GitHub Copilot App [#install-the-github-copilot-app]

    Download it from [github.com/features/ai/github-app](https://github.com/features/ai/github-app) (macOS, Windows, or Linux) and sign in with your GitHub account.
  </Step>

  <Step>
    ### Add LLM Gateway as a Model Provider [#add-llm-gateway-as-a-model-provider]

    In the app, open **Settings** → **Model Providers**, select **Add provider**, and choose the **OpenAI-compatible** provider type. Set the **Base URL** to:

    ```txt
    https://api.llmgateway.io/v1
    ```

    Paste your LLM Gateway API key and save. The key is stored in your OS keychain.
  </Step>

  <Step>
    ### Start a Session with Any Model [#start-a-session-with-any-model]

    LLM Gateway's models now appear in the model picker alongside Copilot-hosted models. Choose one when you start an agent session — each session can use a different model.
  </Step>
</Steps>

## Why This Works [#why-this-works]

LLM Gateway's `/v1` endpoint is fully OpenAI-compatible. The Copilot app fetches the model list from the gateway and routes each agent session through it, and we route requests to the right provider behind the scenes. This means:

* **Use any tool-calling model** — Claude, Gemini, GPT, and the rest of the catalog in Copilot agent sessions
* **Keep your workflow** — sessions, workspaces, and PR merging work exactly the same
* **Track costs** — every request appears in your LLM Gateway dashboard
* **Automatic caching** — repeated requests hit cache, saving money

## Choosing Models [#choosing-models]

All models available to your account show up in the app's model picker; agent sessions work with models that support tool calling and streaming. Browse the [models page](https://llmgateway.io/models) to compare capabilities and pricing, or check [discounted models](https://llmgateway.io/models?discounted=true) for savings up to 90%.

## Good to Know [#good-to-know]

* **Keys stay local** — the app stores your API key in the OS keychain and never reads it back into the UI.
* **Any plan works** — BYOK providers work on every Copilot plan, including Free. You don't need a paid Copilot subscription to run agent sessions through LLM Gateway.
* **Business and Enterprise** — adding model providers is gated by the **Enable custom models** (BYOK) policy, which your admin must turn on. Accessing the Copilot app itself also requires the Copilot CLI enabled in policy settings.
* **Agent sessions only** — BYOK covers the app's model-powered agent sessions. Inline code completions in your editor still use Copilot's own service.

## GitHub Copilot in VS Code [#github-copilot-in-vs-code]

Copilot Chat in VS Code supports custom endpoints too:

1. Run **Chat: Manage Language Models** from the Command Palette
2. Choose **Add Models** → **Custom Endpoint**, enter your LLM Gateway API key, and select **Chat Completions** as the API type (LLM Gateway is OpenAI-compatible)
3. In the generated `chatLanguageModels.json`, point the model `url` at `https://api.llmgateway.io/v1/chat/completions` and set the model `id` from the [models page](https://llmgateway.io/models)

Your gateway models then appear in the VS Code chat model picker.

## Troubleshooting [#troubleshooting]

### Models don't appear in the picker [#models-dont-appear-in-the-picker]

1. Verify the Base URL is exactly `https://api.llmgateway.io/v1` (note the `/v1` at the end)
2. Check your API key starts with `llmgtwy_` and is active in your [dashboard](https://llmgateway.io/dashboard)

### 401 Unauthorized [#401-unauthorized]

Your API key is invalid or was revoked. Generate a new key in the dashboard and update the provider entry in **Settings** → **Model Providers**.

### 402 or credit errors [#402-or-credit-errors]

Your LLM Gateway organization is out of credits. Top up in the [dashboard](https://llmgateway.io/dashboard) — BYOK sessions bill through LLM Gateway, not Copilot premium requests.

### Provider option is missing [#provider-option-is-missing]

BYOK in the Copilot app shipped in June 2026 — update to the latest app version. On Business or Enterprise plans, ask your admin to enable the **Enable custom models** (BYOK) policy — and the Copilot CLI policy if you can't access the app at all.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Hermes Agent Integration
URL: https://llmgateway.io/guides/hermes-agent


[Hermes Agent](https://github.com/nousresearch/hermes-agent) is an open-source AI coding agent for your terminal built by Nous Research. It supports tool use, browser automation, multi-provider routing, skills, and MCP servers. By pointing it at LLM Gateway you get access to 200+ models from 40+ providers, all tracked in one dashboard.

One config change. No code changes. Full cost tracking.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* Hermes Agent installed — see [installation](#installation) below or visit the [Hermes Agent repo](https://github.com/nousresearch/hermes-agent)
* An LLM Gateway API key — [sign up free](https://llmgateway.io/signup) (no credit card required)

## Installation [#installation]

Install Hermes Agent using the official install script:

```bash
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
```

After installation, reload your shell and verify:

```bash
source ~/.bashrc
hermes --version
```

<Callout type="info">
  The installer handles Python 3.11, Node.js, ripgrep, and other dependencies
  automatically. See the [repo](https://github.com/nousresearch/hermes-agent)
  for Windows (PowerShell) and manual install options.
</Callout>

## Setup [#setup]

<Steps>
  <Step>
    ### Run the Setup Wizard [#run-the-setup-wizard]

    Run `hermes setup` to launch the interactive setup wizard. You can choose either **Quick setup** (option 1) for provider, model, and messaging configuration, or **Full setup** (option 2) to configure everything including tools, skills, and advanced options:

    ```bash
    hermes setup
    ```

        <img alt="Hermes Agent Setup Wizard" src="__img0" />

    In this guide we use Quick setup, but Full setup works the same way — it just includes additional configuration steps.
  </Step>

  <Step>
    ### Configure Inference Provider [#configure-inference-provider]

    The wizard will ask you to configure your inference provider. Select **Custom OpenAI-compatible endpoint** and enter the LLM Gateway base URL:

    ```
    API base URL: https://api.llmgateway.io/v1
    ```

    Then paste your LLM Gateway API key (starts with `llmgtwy_`):

        <img alt="Inference Provider Configuration" src="__img1" />
  </Step>

  <Step>
    ### Choose a Model [#choose-a-model]

    The wizard presents a list of 200+ available models. Type a model name or select from the list. Popular choices include `claude-sonnet-4-6`, `gpt-5.5`, or `gemini-3.1-pro`:

        <img alt="Model Selection List" src="__img2" />
  </Step>

  <Step>
    ### Set Context Length [#set-context-length]

    Leave the context length blank to auto-detect (recommended), or specify a custom value:

        <img alt="Context Length Configuration" src="__img3" />
  </Step>

  <Step>
    ### Set Display Name [#set-display-name]

    Give your provider configuration a display name. This appears in the Hermes status bar when chatting:

        <img alt="Display Name Configuration" src="__img4" />
  </Step>

  <Step>
    ### Select Terminal Backend [#select-terminal-backend]

    Choose your terminal backend. In this guide we use **Local** (run directly on this machine), but you can pick any option based on your requirements — Docker for isolated containers, SSH for remote machines, Modal for serverless sandboxes, Daytona for cloud dev environments, and more:

        <img alt="Terminal Backend Selection" src="__img5" />
  </Step>

  <Step>
    ### Setup Complete [#setup-complete]

    Once done, Hermes shows you where your config files are stored and how to edit them. It will prompt &#x2A;*"Launch hermes chat now? \[Y/n]"** — press `Y` to start an interactive agent session immediately:

        <img alt="Setup Complete" src="__img6" />

    Your configuration files:

    * **Settings:** `~/.hermes/config.yaml`
    * **API Keys:** `~/.hermes/.env`
    * **Data:** `~/.hermes/cron/`, `sessions/`, `logs/`

    Once you press `Y`, Hermes launches a full agent session connected to LLM Gateway. You can start chatting right away.
  </Step>
</Steps>

## DevPass Compatibility [#devpass-compatibility]

Hermes Agent is fully compatible with DevPass coding plans. The gateway automatically detects Hermes via multiple signals:

* **X-Source header** — Hermes sends `X-Source: https://hermes-agent.nousresearch.com` (auto-detected)
* **User-Agent** — `HermesAgent/<version>` is recognized
* **X-Title** — Title containing "hermes agent" is matched
* **HTTP-Referer** — Any referer URL containing `hermes-agent.nousresearch.com`

No configuration is needed on your side — DevPass plans automatically allow Hermes traffic.

<Callout type="info">
  Native LLM Gateway provider support is being added to Hermes Agent upstream.
  Once merged, you'll be able to select "LLM Gateway" directly as a provider in
  `hermes setup` instead of using "Custom OpenAI-compatible endpoint".
</Callout>

## Using Hermes with LLM Gateway [#using-hermes-with-llm-gateway]

Once configured, all requests route through LLM Gateway. You'll see the provider name (e.g., "LLMGATEWAY") in the Hermes status bar.

### Switching Models at Runtime [#switching-models-at-runtime]

You can switch models mid-session using the `/model` slash command (similar to how Claude Code uses slash commands). Just type `/model` followed by the model name:

<img alt="Switching to Claude Haiku via LLM Gateway" src="__img7" />

Switch to any model available through LLM Gateway — from Claude to GPT to open-source models — without leaving your session:

<img alt="Switching to GPT-5.4-nano via LLM Gateway" src="__img8" />

Add `--global` to persist the model change across sessions.

### CLI Model Override [#cli-model-override]

You can also override the model from the command line:

```bash
# Use a specific model for this session
hermes chat --model gpt-5.5

# Use a powerful model for complex tasks
hermes chat --model claude-opus-4-6
```

## Why Use LLM Gateway with Hermes Agent [#why-use-llm-gateway-with-hermes-agent]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://llmgateway.io/models?discounted=true) for savings up to 90%

## One-Shot Mode [#one-shot-mode]

For scripting or CI pipelines, use the `-q` flag for a one-shot prompt:

```bash
hermes chat -q "Explain what this function does" -Q
```

The `-Q` flag enables quiet mode, suppressing the banner and spinner for clean output. For pure one-shot mode (no interactive session):

```bash
hermes chat -z "Generate a README for this project"
```

## Useful Hermes Commands [#useful-hermes-commands]

| Command                | Purpose                                 |
| ---------------------- | --------------------------------------- |
| `hermes`               | Start interactive chat (default)        |
| `hermes setup`         | Run the setup wizard                    |
| `hermes setup model`   | Change model/provider                   |
| `hermes chat -q "..."` | One-shot prompt                         |
| `hermes model`         | Choose provider and model interactively |
| `hermes config edit`   | Open config in your editor              |
| `hermes doctor`        | Diagnose connection/config issues       |
| `hermes sessions`      | Browse and manage past sessions         |
| `hermes --continue`    | Resume most recent session              |
| `hermes update`        | Update to latest version                |

## Locking to a Specific Provider [#locking-to-a-specific-provider]

By default, LLM Gateway automatically fails over to alternative providers if your chosen provider is experiencing downtime. To disable fallback and always route to one provider, you can add the header via Hermes's request configuration.

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.llmgateway.io/features/routing) for details.
</Callout>

## Troubleshooting [#troubleshooting]

### Model not found [#model-not-found]

If you get a "model not supported" error, check that your model ID matches exactly what's listed on the [models page](https://llmgateway.io/models). Model IDs are case-sensitive.

### Connection timeout [#connection-timeout]

Verify your `base_url` is set to `https://api.llmgateway.io/v1` (note the `/v1` at the end). You can also check the `HERMES_API_TIMEOUT` environment variable if you're hitting timeouts on long-running requests.

### Authentication errors [#authentication-errors]

Make sure your `api_key` starts with `llmgtwy_` and is valid. Check your [dashboard](https://llmgateway.io/dashboard) to confirm the key is active.

### Diagnosing issues [#diagnosing-issues]

Run `hermes doctor` to check your configuration, connectivity, and credentials:

```bash
hermes doctor
```

### Old config overrides [#old-config-overrides]

If you previously used a different provider (e.g., OpenRouter), make sure to update both `provider` and `base_url` fields. The `provider` must be set to `"custom"` for LLM Gateway. Also check `~/.hermes/.env` for any leftover `OPENROUTER_API_KEY` or other provider keys that might take precedence.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Kilo Code Integration
URL: https://llmgateway.io/guides/kilo-code


[Kilo Code](https://kilo.ai/) is an AI coding assistant that runs as a VS Code extension. It supports autonomous coding, file editing, terminal commands, and browser automation. LLM Gateway is a built-in provider in Kilo Code, so setup takes under a minute — no manual base URL configuration required.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* VS Code or a VS Code-based editor (Cursor, Windsurf, etc.)
* An LLM Gateway API key — [sign up free](https://llmgateway.io/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install Kilo Code [#install-kilo-code]

    Open VS Code, go to the Extensions view (Ctrl+Shift+X / Cmd+Shift+X), search for **Kilo Code**, and click **Install**.

    Alternatively, install from the [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=kilocode.kilo-code).
  </Step>

  <Step>
    ### Open Providers Settings [#open-providers-settings]

    Click the Kilo Code icon in the VS Code sidebar, then open **Settings > Providers**. You'll see the list of popular providers:

        <img alt="Kilo Code Providers screen" src="__img0" />
  </Step>

  <Step>
    ### Find LLM Gateway [#find-llm-gateway]

    Click **Show more providers** at the bottom of the list. In the "Connect provider" dialog, type `llm` in the search box — **LLM Gateway** will appear:

        <img alt="Searching for LLM Gateway" src="__img1" />

    Click the &#x2A;*+** button next to LLM Gateway.
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    Kilo Code will show the **Connect LLM Gateway** dialog. Paste your LLM Gateway API key (starts with `llmgtwy_`) and click **Submit**:

        <img alt="Connect LLM Gateway — enter API key" src="__img2" />

    [Sign up](https://llmgateway.io/signup) or log in to your LLM Gateway dashboard and navigate to **API Keys** to get your key.
  </Step>

  <Step>
    ### Start Coding [#start-coding]

    Once connected, select an LLM Gateway model from the model picker at the bottom of the chat panel. All requests now route through LLM Gateway — you'll see usage, costs, and logs in your [dashboard](https://llmgateway.io/dashboard):

        <img alt="Kilo Code chat active with LLM Gateway" src="__img3" />
  </Step>
</Steps>

## Why Use LLM Gateway with Kilo Code [#why-use-llm-gateway-with-kilo-code]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more from 40+ providers
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://llmgateway.io/models?discounted=true) for savings up to 90%

## Features [#features]

Once configured, you can use all of Kilo Code's features with LLM Gateway:

* **Autonomous coding** — Create and edit files, build features from natural language
* **Terminal commands** — Run builds, tests, and scripts directly from the chat
* **Browser automation** — Preview and interact with web apps
* **Checkpoints** — Save and restore session states
* **Multiple modes** — Switch between Code, Architect, Ask, and Debug modes

## Switching Models [#switching-models]

Click the model name at the bottom of the Kilo Code chat panel to open the model picker. Select any LLM Gateway model — the switch takes effect immediately for the next message.

## Troubleshooting [#troubleshooting]

### LLM Gateway not in provider list [#llm-gateway-not-in-provider-list]

Click **Show more providers** at the bottom of the Providers page. In the search dialog, type "llm" or "gateway" to find it.

### Authentication errors [#authentication-errors]

Make sure your API key starts with `llmgtwy_` and is active. Check your [dashboard](https://llmgateway.io/dashboard) to confirm the key is valid.

### Model not found [#model-not-found]

Verify the model ID matches exactly what's listed on the [models page](https://llmgateway.io/models). Model IDs are case-sensitive.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Kimi Code Integration
URL: https://llmgateway.io/guides/kimi-code


[Kimi Code CLI](https://github.com/MoonshotAI/kimi-code) is an open-source, AI-powered coding agent developed by Moonshot AI designed to automate software development tasks directly within your terminal. It can read and edit code, execute shell commands, search files, and autonomously manage complex coding workflows.

Kimi Code features first-class support for the **models.dev** registry, a community-maintained model catalog. This allows Kimi Code to query and configure LLM Gateway dynamically — fetching all compatible models, capabilities (such as thinking or vision), and pricing without requiring manual TOML editing.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* An LLM Gateway API key — [sign up free](https://llmgateway.io/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install Kimi Code CLI [#install-kimi-code-cli]

    If you haven't already, install Kimi Code CLI.

    * **macOS or Linux**:

      ```bash
      curl -fsSL https://code.kimi.com/kimi-code/install.sh | bash
      ```

    * **Homebrew (macOS/Linux)**:

      ```bash
      brew install kimi-code
      ```

    * **Windows (PowerShell)**:
      ```powershell
      irm https://code.kimi.com/kimi-code/install.ps1 | iex
      ```

    Confirm the installation:

    ```bash
    kimi --version
    ```
  </Step>

  <Step>
    ### Launch Kimi Code and Open the Provider Manager [#launch-kimi-code-and-open-the-provider-manager]

    Start the interactive terminal in your project directory:

    ```bash
    kimi
    ```

    Once loaded, type the `/provider` command and press Enter. Select **Known third-party provider** to fetch the catalog from the registry:

    <img src="/guides/kimi-code/0-add-provider.png" alt="Opening Provider Manager in Kimi Code" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>

  <Step>
    ### Select LLM Gateway [#select-llm-gateway]

    Type `llm` to filter the providers and select **LLM Gateway** from the list:

    <img src="/guides/kimi-code/1-select-provider.png" alt="Selecting LLM Gateway in Kimi Code" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    When prompted, paste your LLM Gateway API key and press Enter. Kimi Code will save it securely to your local configuration:

    <img src="/guides/kimi-code/2-enter-key.png" alt="Entering LLM Gateway API Key" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    <Callout type="info">
      Your credentials are saved locally to `~/.kimi-code/config.toml`.
    </Callout>
  </Step>

  <Step>
    ### Select a Model and Toggle Thinking [#select-a-model-and-toggle-thinking]

    The LLM Gateway catalog is now loaded. Use the arrow keys to browse or type to search for your desired model. Select the `llmgateway` tab to view only LLM Gateway models.

    You can also toggle the **Thinking** option (On/Off) at the bottom depending on the model's capabilities:

    <img src="/guides/kimi-code/3-select-model.png" alt="Browsing LLM Gateway Models" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    For example, type `gpt-5.5` to find the latest reasoning model, select it, and press Enter:

    <img src="/guides/kimi-code/4-select-gpt-model.png" alt="Selecting GPT-5.5 Model" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>

  <Step>
    ### Start Coding [#start-coding]

    All set! Kimi Code is now configured. Your requests will be securely routed through LLM Gateway, allowing you to use advanced models for local autonomous coding while showing real-time usage and cost statistics on your LLM Gateway dashboard.

    <img src="/guides/kimi-code/5-chat.png" alt="Running Kimi Code CLI session" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    Use `/model` in the terminal session at any time to switch models.
  </Step>
</Steps>

## Manual Configuration (Advanced) [#manual-configuration-advanced]

If you prefer to configure your environment manually without using the interactive provider manager, you can write settings directly to your configuration file at `~/.kimi-code/config.toml` (or `C:\Users\<YourUsername>\.kimi-code\config.toml` on Windows).

Here is an example TOML configuration that registers **GPT-5.5**, **Claude 3.7 Sonnet**, **DeepSeek R1**, and **Qwen3.7 Max** manually:

```toml
default_model = "llmgateway/gpt-5.5"

[providers.llmgateway]
type = "openai"
api_key = "llmgtwy_your_api_key_here"
base_url = "https://api.llmgateway.io/v1"

[models."llmgateway/gpt-5.5"]
provider = "llmgateway"
model = "gpt-5.5"
max_context_size = 1050000
max_output_size = 128000
capabilities = [ "thinking", "tool_use" ]
display_name = "GPT-5.5"

[models."llmgateway/claude-3.7-sonnet"]
provider = "llmgateway"
model = "claude-3.7-sonnet"
max_context_size = 200000
max_output_size = 8192
capabilities = [ "image_in", "thinking", "tool_use" ]
display_name = "Claude 3.7 Sonnet"

[models."llmgateway/deepseek-r1"]
provider = "llmgateway"
model = "deepseek-r1"
max_context_size = 131072
max_output_size = 8192
capabilities = [ "thinking", "tool_use" ]
display_name = "DeepSeek R1"

[models."llmgateway/qwen3.7-max"]
provider = "llmgateway"
model = "qwen3.7-max"
max_context_size = 1000000
max_output_size = 65536
capabilities = [ "thinking", "tool_use" ]
display_name = "Qwen3.7 Max"
```

## Why Use LLM Gateway with Kimi Code CLI [#why-use-llm-gateway-with-kimi-code-cli]

* **200+ models** — Access GPT-5.5, Gemini, Llama, DeepSeek, and more in a single CLI configuration.
* **Unified cost tracking** — Get a detailed breakdown of costs per prompt and session in your dashboard.
* **Response caching** — Automatically cache repeated requests (such as parsing or building commands) to save API costs.
* **Automatic fallback** — Keep coding even if a provider encounters temporary downtime.
* **Volume discounts** — Access selected models with up to 90% savings compared to standard pricing.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# MiMo Code Integration
URL: https://llmgateway.io/guides/mimocode


[MiMo Code](https://mimo.xiaomi.com/mimocode) is an AI-powered coding agent command-line tool developed by Xiaomi. It can understand your code repository, plan changes, safely execute shell commands, edit files, and autonomously manage complex software development tasks in your terminal.

By configuring MiMo Code to route through LLM Gateway, you can point it at any model—GPT-5.5, Gemini, Llama, Claude, or 210+ others—while keeping the same API format MiMo Code expects, with full cost tracking in your dashboard.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* An LLM Gateway API key — [sign up free](https://llmgateway.io/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install MiMo Code [#install-mimo-code]

    If you haven't already, install MiMo Code by running the official installation command in your terminal:

    ```bash
    curl -fsSL https://mimo.xiaomi.com/install | bash
    ```

    Confirm the installation by checking the help command:

    ```bash
    mimo --help
    ```
  </Step>

  <Step>
    ### Configure mimocode.json [#configure-mimocodejson]

    Create or edit your MiMo Code configuration file at `~/.config/mimocode/mimocode.json` (on Linux/macOS) or `~/.mimocode/mimocode.json`.

    Specify the default models you want to use and route the `anthropic` provider to your LLM Gateway endpoint. Here is an example configuration that sets up **Claude Opus 4.8**, **GPT-5.5**, **DeepSeek V4 Pro**, **MiniMax M3**, and **Qwen3.7 Max**:

    ```json
    {
    	"model": "anthropic/claude-opus-4-8",
    	"small_model": "anthropic/claude-4-5-haiku-latest",
    	"provider": {
    		"anthropic": {
    			"options": {
    				"apiKey": "llmgtwy_your_api_key_here",
    				"baseURL": "https://api.llmgateway.io/v1"
    			},
    			"models": {
    				"gpt-5.5": {
    					"name": "gpt-5.5"
    				},
    				"claude-opus-4-8": {
    					"name": "claude-opus-4-8"
    				},
    				"deepseek-v4-pro": {
    					"name": "deepseek-v4-pro"
    				},
    				"minimax-m3": {
    					"name": "minimax-m3"
    				},
    				"qwen3.7-max": {
    					"name": "qwen3.7-max"
    				}
    			}
    		}
    	}
    }
    ```

    <img src="/guides/mimocode/0-config.png" alt="Configuring mimocode.json" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    <Callout type="info">
      Replace `llmgtwy_your_api_key_here` with your actual LLM Gateway API key from
      the dashboard.
    </Callout>
  </Step>

  <Step>
    ### Alternatively: Use Environment Variables [#alternatively-use-environment-variables]

    If you prefer to configure the provider dynamically, you can export the standard Anthropic environment variables before starting MiMo Code:

    ```bash
    export ANTHROPIC_API_KEY=llmgtwy_your_api_key_here
    export ANTHROPIC_BASE_URL=https://api.llmgateway.io/v1
    ```
  </Step>

  <Step>
    ### Run MiMo Code [#run-mimo-code]

    Navigate to your project folder and launch the TUI or run a prompt directly:

    ```bash
    mimo
    ```

    Or run it with a message:

    ```bash
    mimo run "Your coding prompt here"
    ```

    All requests will now be routed through LLM Gateway, allowing you to use advanced models for local autonomous coding while showing real-time usage and cost statistics on your LLM Gateway dashboard.

    <img src="/guides/mimocode/1-chat.png" alt="Running MiMo Code with LLM Gateway" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>
</Steps>

## Configuration Details [#configuration-details]

### The Provider Options [#the-provider-options]

To point MiMo Code to LLM Gateway, you define the `baseURL` and `apiKey` inside the `options` of the `anthropic` provider block.

```json
"provider": {
	"anthropic": {
		"options": {
			"apiKey": "llmgtwy_your_api_key_here",
			"baseURL": "https://api.llmgateway.io/v1"
		}
	}
}
```

### Defining Custom Models [#defining-custom-models]

Because MiMo Code CLI restricts requests to built-in models by default, any custom model you wish to target (such as `gpt-5.5` or `deepseek-v4-pro`) must be registered in the `models` dictionary within the `anthropic` provider config:

```json
"models": {
	"gpt-5.5": {
		"name": "gpt-5.5"
	}
}
```

Once registered, you can set them as your default model or small model using the `anthropic/` prefix (e.g. `"model": "anthropic/gpt-5.5"`).

## Why Use LLM Gateway with MiMo Code [#why-use-llm-gateway-with-mimo-code]

* **200+ models** — Access GPT-5.5, Gemini, Llama, DeepSeek, and more in a single CLI configuration.
* **Unified cost tracking** — Get a detailed breakdown of costs per prompt and session in your dashboard.
* **Response caching** — Automatically cache repeated requests (such as parsing or building commands) to save API costs.
* **Automatic fallback** — Keep coding even if a provider encounters temporary downtime.
* **Volume discounts** — Access selected models with up to 90% savings compared to standard pricing.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# N8n Integration
URL: https://llmgateway.io/guides/n8n


n8n is a powerful workflow automation tool that can be enhanced with AI capabilities through LLM Gateway. This guide shows how to integrate LLM Gateway into your n8n workflows.

<img alt="n8n workflow with LLM Gateway" src="__img0" />

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* An LLM Gateway account with an API key
* n8n instance (self-hosted or cloud)
* Basic understanding of n8n workflows

## Setup [#setup]

The easiest way to use LLM Gateway with n8n is through the OpenAI node with custom configuration.

<Steps>
  <Step>
    ### Add OpenAI Credentials [#add-openai-credentials]

    1. In n8n, go to **Settings** → **Credentials**

        <img alt="n8n credentials" src="__img1" />

    2. Click **Add Credential** → **OpenAI**

        <img alt="n8n credentials" src="__img2" />

    3. Configure as follows:
       * **API Key**: Your LLM Gateway API key
       * **Base URL**: `https://api.llmgateway.io/v1`
       * **Organization ID**: Leave blank

        <img alt="n8n credentials" src="__img3" />
  </Step>

  <Step>
    ### Configure OpenAI Node [#configure-openai-node]

    1. Add an **AI Agent** node to your workflow
    2. Add a **Chat Model** edge to the node

        <img alt="n8n credentials" src="__img4" />

    3. Configure the node to use the LLMGateway provider

        <img alt="n8n credentials" src="__img5" />

    <Callout type="warning">
      Note: You have to toggle off the responses API. LLMGateway does not support
      it.
    </Callout>

        <img alt="responses api" src="__img6" />

    4. Select your desired options

    * **Model**: Use any [LLMGateway model](https://llmgateway.io/models) ID (e.g., `gpt-5`)
    * **Options**: Optionally, configure LLM parameters

        <img alt="n8n credentials" src="__img7" />
  </Step>

  <Step>
    ### Test Workflow [#test-workflow]

    Finally, try running your workflow with a test prompt.

        <img alt="n8n credentials" src="__img8" />
  </Step>
</Steps>


# OpenClaw Integration
URL: https://llmgateway.io/guides/openclaw


[OpenClaw](https://docs.openclaw.ai/) is a self-hosted gateway that connects your favorite chat apps—WhatsApp, Telegram, Discord, iMessage, and more—to AI coding agents. With LLM Gateway as a custom provider, you can route all your OpenClaw traffic through a single API, use any of 200+ models, and keep full visibility into usage and costs.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Keep the `llmgateway/`
  prefix in your OpenClaw config — it names OpenClaw's local provider entry
  (e.g. `llmgateway/gpt-5.4`). But the model ID after it must be a canonical ID
  without a gateway provider prefix (`llmgateway/claude-sonnet-4-5`, not
  `llmgateway/anthropic/claude-sonnet-4-5`) — provider-pinned routing is not
  available on coding plans; the gateway picks the provider for you.
</Callout>

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for LLM Gateway [#sign-up-for-llm-gateway]

    [Sign up free](https://llmgateway.io/signup) — no credit card required. Create an API key in the dashboard and copy it when shown.
  </Step>

  <Step>
    ### Set Your API Key [#set-your-api-key]

    ```bash
    export LLMGATEWAY_API_KEY=llmgtwy_your_api_key_here
    ```
  </Step>

  <Step>
    ### Configure OpenClaw [#configure-openclaw]

    Add LLM Gateway as a custom provider in your `~/.openclaw/openclaw.json`:

    ```json
    {
    	"models": {
    		"mode": "merge",
    		"providers": {
    			"llmgateway": {
    				"baseUrl": "https://api.llmgateway.io/v1",
    				"apiKey": "${LLMGATEWAY_API_KEY}",
    				"api": "openai-completions",
    				"models": [
    					{
    						"id": "gpt-5.4",
    						"name": "GPT-5.4",
    						"contextWindow": 128000,
    						"maxTokens": 32000
    					},
    					{
    						"id": "claude-opus-4-6",
    						"name": "Claude Opus 4.6",
    						"contextWindow": 200000,
    						"maxTokens": 8192
    					},
    					{
    						"id": "gemini-3.1-pro-preview",
    						"name": "Gemini 3.1 Pro",
    						"contextWindow": 1000000,
    						"maxTokens": 8192
    					}
    				]
    			}
    		}
    	},
    	"agents": {
    		"defaults": {
    			"model": {
    				"primary": "llmgateway/gpt-5.4"
    			}
    		}
    	}
    }
    ```
  </Step>

  <Step>
    ### Start Chatting [#start-chatting]

    Launch OpenClaw and start chatting across your connected channels. All requests will be routed through LLM Gateway.
  </Step>
</Steps>

## Why Use LLM Gateway with OpenClaw [#why-use-llm-gateway-with-openclaw]

* **Model flexibility** — Switch between GPT-5.4, Claude Opus, Gemini, or any of 200+ models
* **Cost tracking** — Monitor exactly how much your chat agents cost to run
* **Single bill** — No need to manage multiple API provider accounts
* **Response caching** — Repeated queries hit cache, reducing costs
* **Rate limit handling** — Automatic fallback between providers

## Switching Models [#switching-models]

Change the primary model in your config to switch between any model:

```json
{
	"agents": {
		"defaults": {
			"model": { "primary": "llmgateway/claude-opus-4-6" }
		}
	}
}
```

## Model Fallback Chain [#model-fallback-chain]

OpenClaw supports fallback models. If the primary model is unavailable, it automatically falls back:

```json
{
	"agents": {
		"defaults": {
			"model": {
				"primary": "llmgateway/gpt-5.4",
				"fallbacks": ["llmgateway/claude-opus-4-6"]
			}
		}
	}
}
```

## Available Models [#available-models]

LLM Gateway uses canonical model IDs with smart routing—automatically selecting the best provider based on uptime, throughput, price, and latency. You can use any model from the [models page](https://llmgateway.io/models). Flagship models include:

| Model                    | Best For                                    |
| ------------------------ | ------------------------------------------- |
| `gpt-5.4`                | Latest OpenAI flagship, highest quality     |
| `claude-opus-4-6`        | Anthropic's most capable model              |
| `claude-sonnet-4-6`      | Fast reasoning with extended thinking       |
| `gemini-3.1-pro-preview` | Google's latest flagship, 1M context window |
| `o3`                     | Advanced reasoning tasks                    |
| `gpt-5.4-pro`            | Premium tier with extended reasoning        |
| `gemini-3.6-flash`       | Fast responses, good for high-volume        |
| `claude-haiku-4-5`       | Cost-effective, quick responses             |
| `grok-3`                 | xAI flagship                                |
| `deepseek-v3.1`          | Open-source with vision and tools           |

For more details on routing behavior, see [routing](https://docs.llmgateway.io/features/routing).

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

## Tips for Chat Agents [#tips-for-chat-agents]

### Optimize Costs [#optimize-costs]

1. **Use smaller models for simple tasks** — Claude Haiku or Gemini Flash handle basic Q\&A well
2. **Enable caching** — LLM Gateway caches identical requests automatically
3. **Set token limits** — Configure max tokens to prevent runaway costs

### Improve Response Quality [#improve-response-quality]

1. **Choose the right model** — Claude Opus excels at nuanced conversation, GPT-5.4 at general tasks
2. **Use system prompts** — Configure your agent's personality and capabilities
3. **Test multiple models** — LLM Gateway makes it easy to A/B test different providers

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# OpenCode Desktop Integration
URL: https://llmgateway.io/guides/opencode-desktop


[OpenCode Desktop](https://opencode.ai/download) is the GUI desktop app version of OpenCode — an open-source AI coding agent with a full visual interface for managing providers, models, and sessions. LLM Gateway is a built-in provider, so setup takes under a minute with no config files required.

<Callout type="info">
  Looking for the CLI version? See the [OpenCode CLI guide](https://docs.llmgateway.io/guides/opencode).
</Callout>

## Two Built-In Providers [#two-built-in-providers]

OpenCode ships two LLM Gateway entries, sharing one endpoint and one API key. **LLM Gateway** lists provider-pinned models (`anthropic/claude-opus-5`, named "Claude Opus 5 (Anthropic)") for pay-as-you-go keys; &#x2A;*DevPass (LLM Gateway)** lists canonical models (`claude-opus-5`) that the gateway routes for you. See the [CLI guide](https://docs.llmgateway.io/guides/opencode#two-built-in-providers) for the full comparison.

<Callout type="info">
  **Using DevPass?** Connect &#x2A;*DevPass (LLM Gateway)**. Provider-pinned routing
  is not available on coding plans, so canonical model IDs are the ones that
  work.
</Callout>

## Prerequisites [#prerequisites]

* OpenCode Desktop installed — [download for Windows or macOS](https://opencode.ai/download)
* An LLM Gateway API key — [sign up free](https://llmgateway.io/signup) (no credit card required)

## Installation [#installation]

Download OpenCode Desktop from [opencode.ai/download](https://opencode.ai/download) and install it for your platform:

* **macOS (Apple Silicon)** — `.dmg` installer
* **macOS (Intel)** — `.dmg` installer
* **Windows** — `.exe` installer

You can also install on macOS via Homebrew:

```bash
brew install --cask opencode-desktop
```

## Setup [#setup]

<Steps>
  <Step>
    ### Open Providers Settings [#open-providers-settings]

    Launch OpenCode Desktop. Click the **Providers** section in the left sidebar under **Server**. You'll see the list of built-in providers:

        <img alt="OpenCode Desktop Providers screen" src="__img0" />
  </Step>

  <Step>
    ### Find LLM Gateway [#find-llm-gateway]

    Click **Show more providers*&#x2A; at the bottom of the list, or click **+ Connect** on any entry to open the provider search. Type `LLM` in the search box — both **LLM Gateway** and &#x2A;*DevPass (LLM Gateway)** appear under "Other":

        <img alt="Searching for LLM Gateway" src="__img1" />

    Select the one matching your key. You can connect both — they take the same key, and each contributes its models to the picker.
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    OpenCode will show the **Connect LLM Gateway** dialog. Paste your LLM Gateway API key (starts with `llmgtwy_`) and click **Continue**:

        <img alt="Connect LLM Gateway — enter API key" src="__img2" />

    [Sign up](https://llmgateway.io/signup) or log in to your LLM Gateway dashboard and navigate to **API Keys** to get your key.
  </Step>

  <Step>
    ### Select a Model [#select-a-model]

    Once connected, open the model picker from the chat input bar. Type `llm` to filter LLM Gateway models. Pinned entries carry their serving provider in the name — "Claude Opus 5 (Anthropic)", "GPT-5.5 (Azure)" — while DevPass entries show the canonical name:

        <img alt="LLM Gateway model selection" src="__img3" />
  </Step>

  <Step>
    ### Start Building [#start-building]

    Select a model and start chatting. All requests route through LLM Gateway — you'll see usage, costs, and logs in your [dashboard](https://llmgateway.io/dashboard):

        <img alt="OpenCode Desktop chat active with LLM Gateway" src="__img4" />
  </Step>
</Steps>

## Why Use LLM Gateway with OpenCode Desktop [#why-use-llm-gateway-with-opencode-desktop]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more from 40+ providers
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://llmgateway.io/models?discounted=true) for savings up to 90%

## Switching Models [#switching-models]

You can switch models at any time from the model picker in the chat input bar. Click the current model name, type `llm` to filter to LLM Gateway models, and select a new one. The switch takes effect immediately for the next message.

## Locking to a Specific Provider [#locking-to-a-specific-provider]

Pick a model from the **LLM Gateway** entry to choose its upstream provider. LLM Gateway still fails over to an alternative provider if that one is down; to make the pin strict, pass the `X-No-Fallback` header via a custom `opencode.json` in your project root:

```json
{
	"provider": {
		"llmgateway-providers": {
			"options": {
				"headers": {
					"X-No-Fallback": "true"
				}
			}
		}
	}
}
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.llmgateway.io/features/routing) for details.
</Callout>

## Troubleshooting [#troubleshooting]

### LLM Gateway doesn't appear in provider list [#llm-gateway-doesnt-appear-in-provider-list]

Click **Show more providers** at the bottom of the Providers page to expand the full list, then search for "LLM".

### Authentication errors [#authentication-errors]

Make sure your API key starts with `llmgtwy_` and is active. Check your [dashboard](https://llmgateway.io/dashboard) to confirm the key is valid.

### Models not loading after connect [#models-not-loading-after-connect]

Try disconnecting and reconnecting the provider from Settings > Providers. If models still don't load, check your internet connection and verify the key is valid.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# OpenCode Integration
URL: https://llmgateway.io/guides/opencode


[OpenCode](https://opencode.ai) is an open-source AI coding agent for your terminal, IDE, or desktop. LLM Gateway is built into OpenCode, so setup takes under a minute — no config files or npm adapters required. You get access to 200+ models from 40+ providers, all tracked in one dashboard.

## Two Built-In Providers [#two-built-in-providers]

OpenCode ships two LLM Gateway entries. They share the same endpoint (`https://api.llmgateway.io/v1`), the same `LLMGATEWAY_API_KEY`, and the same dashboard — only the model IDs differ:

| Provider in OpenCode      | Provider ID            | Model IDs                          | Use it for                                                                                                                              |
| ------------------------- | ---------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **LLM Gateway**           | `llmgateway-providers` | `anthropic/claude-opus-5` (pinned) | Pay-as-you-go keys. One entry per upstream deployment, carrying that deployment's own pricing, context/output limits, and capabilities. |
| **DevPass (LLM Gateway)** | `llmgateway`           | `claude-opus-5` (canonical)        | [DevPass](https://devpass.llmgateway.io) plan keys, and pay-as-you-go when you want the gateway to route for you.                       |

Pinned entries name the serving provider — "GPT-5.5 (Azure)" versus "GPT-5.5 (OpenAI)" — so duplicate deployments of the same model stay distinguishable and you can pick on price, region, or capability.

<Callout type="info">
  **Using DevPass?** Pick &#x2A;*DevPass (LLM Gateway)**. Provider-pinned routing is
  not available on coding plans, so canonical model IDs are the ones that work.
</Callout>

<Callout type="info">
  The `llmgateway` provider ID is unchanged — only its display name became
  "DevPass (LLM Gateway)". Existing configs keep working as-is.
</Callout>

## Prerequisites [#prerequisites]

* OpenCode installed — visit the [OpenCode download page](https://opencode.ai/download) for your platform
* An LLM Gateway API key

## Setup [#setup]

<Steps>
  <Step>
    ### Launch OpenCode [#launch-opencode]

    Start OpenCode from your terminal:

    ```bash
    opencode
    ```

    **In VS Code/Cursor:**

    1. Install the OpenCode extension from the marketplace
    2. Open Command Palette (Ctrl+Shift+P or Cmd+Shift+P)
    3. Type "OpenCode" and select "Open opencode"
  </Step>

  <Step>
    ### Open the Provider List [#open-the-provider-list]

    Once OpenCode launches, run the `/providers` or `/connect` command to open the provider selection screen.
  </Step>

  <Step>
    ### Select LLM Gateway [#select-llm-gateway]

    Both entries are listed as built-in providers. Select **LLM Gateway** for pay-as-you-go, or &#x2A;*DevPass (LLM Gateway)** if you have a DevPass plan key. You can connect both — they take the same key, and each contributes its models to the picker.
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    OpenCode will prompt you for your API key. Enter your LLM Gateway API key and press Enter. OpenCode will automatically save your credentials securely.

    [Sign up for LLM Gateway](https://llmgateway.io/signup) and create an API key from your dashboard.
  </Step>

  <Step>
    ### Start Using OpenCode [#start-using-opencode]

    You're all set! OpenCode is now connected to LLM Gateway. You can start asking questions and building with AI.
  </Step>
</Steps>

## Why Use LLM Gateway with OpenCode [#why-use-llm-gateway-with-opencode]

* **200+ models** — GPT-5, Claude, Gemini, Llama, and more from 40+ providers
* **One API key** — Stop juggling credentials for every provider
* **Cost tracking** — See what each coding agent costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Volume discounts** — The more you use, the more you save

## Adding Custom Models [#adding-custom-models]

The built-in providers cover the standard LLM Gateway catalog. If you want to add custom model aliases or a model not yet listed, you can create a `config.json` in your OpenCode configuration directory:

**macOS/Linux:** `~/.config/opencode/config.json`

**Windows:** `C:\Users\YourUsername\.config\opencode\config.json`

```json
{
	"provider": {
		"llmgateway": {
			"models": {
				"deepseek-v3": {
					"name": "DeepSeek V3"
				}
			}
		}
	}
}
```

Both entries are built-in providers, so you only need to specify what you're adding — OpenCode merges your config with the built-in definition, and `npm`, `name`, and `baseURL` don't need to be repeated. Add pinned `provider/model` IDs under `llmgateway-providers` and canonical ones under `llmgateway`.

After updating `config.json`, restart OpenCode to see the new models.

## Pinning a Model to a Specific Provider [#pinning-a-model-to-a-specific-provider]

Pinning is what the **LLM Gateway** (`llmgateway-providers`) entry is for. Its model IDs use LLM Gateway's [`provider/model` syntax](https://docs.llmgateway.io/features/routing#provider-specific-routing), so picking "Claude Opus 5 (Anthropic)" in the model picker sends the request to that upstream deployment instead of letting the gateway choose. The &#x2A;*DevPass (LLM Gateway)** entry lists canonical IDs, which the gateway routes to the best available provider automatically.

On its own the pin is a preference, not a guarantee — if that provider is down or rate-limited, the gateway still fails over to another one. When you need it to hold, such as for a fixed price or a compliance boundary, pair the pin with [strict fallback](#disabling-fallback).

Pin one as your default model, or per-agent:

```json
{
	"model": "llmgateway-providers/anthropic/claude-opus-5"
}
```

Anything the built-in catalog does not cover yet you can add by hand, and it shows up in the picker alongside the rest:

```json
{
	"provider": {
		"llmgateway-providers": {
			"models": {
				"aws-bedrock/claude-sonnet-5": {
					"name": "Claude Sonnet 5 (AWS Bedrock)"
				}
			}
		}
	}
}
```

The [LLM Gateway CLI](https://docs.llmgateway.io/developers/cli#configure) can generate these entries for you, with display names, context limits, and per-provider pricing:

```bash
npx @llmgateway/cli configure opencode
```

For providers with multiple regions, append a region suffix to pin both provider and region, e.g. `aws-bedrock/claude-sonnet-5:us`. See the [routing docs](https://docs.llmgateway.io/features/routing#provider-specific-routing) for the full syntax, and each model's available providers on the [models page](https://llmgateway.io/models).

### Disabling Fallback [#disabling-fallback]

By default, LLM Gateway automatically fails over to alternative providers if your chosen provider is experiencing downtime — even for provider-pinned model IDs. To make the pin strict, pass the `X-No-Fallback` header. Requests will then be sent only to the provider you specified, with no automatic fallback.

```json
{
	"provider": {
		"llmgateway-providers": {
			"options": {
				"headers": {
					"X-No-Fallback": "true"
				}
			}
		}
	}
}
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing
  docs](https://docs.llmgateway.io/features/routing#disabling-fallback-with-x-no-fallback-header) for
  details.
</Callout>

## Switching Models [#switching-models]

Select a different model directly in the OpenCode interface, or update the `model` field in your configuration:

```json
{
	"model": "llmgateway-providers/anthropic/claude-opus-5"
}
```

Drop the provider prefix to let the gateway route instead — `llmgateway/claude-opus-5`.

<Callout type="info">
  View all available models on the [models page](https://llmgateway.io/models).
</Callout>

## Troubleshooting [#troubleshooting]

### Connection timeout [#connection-timeout]

Check that you have an active internet connection and that your API key is valid from the [dashboard](https://llmgateway.io/dashboard).

### "Direct provider routing is not available on coding plans" [#direct-provider-routing-is-not-available-on-coding-plans]

Your key belongs to a DevPass plan, and the model you picked came from the **LLM Gateway** entry, whose IDs are provider-pinned. Switch to a model under &#x2A;*DevPass (LLM Gateway)** instead.

### Custom models not showing up [#custom-models-not-showing-up]

After editing `config.json`, restart OpenCode completely for changes to take effect.

### 404 Not Found errors with custom config [#404-not-found-errors-with-custom-config]

If you are using a custom `config.json`, verify your `baseURL` is set to `https://api.llmgateway.io/v1` (note the `/v1` at the end).

## Configuration Tips [#configuration-tips]

* **Global configuration**: Use `~/.config/opencode/config.json` to apply settings across all projects
* **Project-specific**: Place `opencode.json` in your project root to override global settings for that project
* **Model selection**: You can specify different models for different types of tasks using OpenCode's agent configuration

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>


# Pi Integration
URL: https://llmgateway.io/guides/pi


[Pi](https://pi.dev) is a minimal terminal-based coding agent that gives an AI full access to read, write, edit, and run shell commands in your project. By pointing Pi at LLM Gateway, you can use any of our 200+ models — GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.7, DeepSeek V4, and more — with full cost tracking and caching.

<Callout type="info">
  **Using DevPass?** This integration also works with a
  [DevPass](https://devpass.llmgateway.io) plan key. Use canonical model IDs
  without a provider prefix (`claude-sonnet-4-5`, not
  `anthropic/claude-sonnet-4-5`) — provider-pinned routing is not available on
  coding plans; the gateway picks the provider for you.
</Callout>

## Prerequisites [#prerequisites]

* An LLM Gateway account with an API key
* Pi installed (`curl -fsSL https://pi.dev/install.sh | bash`)
* Basic terminal familiarity

## Setup [#setup]

Pi uses a `models.json` configuration file to define providers and models. We'll add LLM Gateway as a custom provider.

<Steps>
  <Step>
    ### Get Your API Key [#get-your-api-key]

    1. Log in to your [LLM Gateway dashboard](https://llmgateway.io/dashboard)
    2. Navigate to **API Keys** section
    3. Create a new API key and copy the key
  </Step>

  <Step>
    ### Configure Pi [#configure-pi]

    Open (or create) the Pi models configuration file at `~/.pi/agent/models.json` and add LLM Gateway as a provider:

    ```json
    {
    	"providers": {
    		"llmgateway": {
    			"baseUrl": "https://api.llmgateway.io/v1",
    			"api": "openai-completions",
    			"apiKey": "llmgtwy_your_api_key_here",
    			"models": [
    				{ "id": "gpt-5.5", "name": "GPT-5.5" },
    				{ "id": "claude-opus-4-7", "name": "Claude Opus 4.7" },
    				{ "id": "gemini-3.1-pro", "name": "Gemini 3.1 Pro" },
    				{ "id": "deepseek-v4", "name": "DeepSeek V4", "reasoning": true }
    			]
    		}
    	}
    }
    ```

    Replace `llmgtwy_your_api_key_here` with your actual API key from Step 1.

        <img alt="Pi models.json Configuration" src="__img0" />

    <Callout type="info">
      Pi reloads `models.json` when you open the `/model` menu — no restart needed
      after editing.
    </Callout>
  </Step>

  <Step>
    ### Select Your Model [#select-your-model]

    1. Run `pi` in any project directory
    2. Type `/model` to open the model selector
    3. Select your LLM Gateway model from the list

    All requests now route through LLM Gateway with full cost tracking.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    Ask Pi to do something in your project to verify everything works:

    ```
    > hello
    ```

        <img alt="Pi Test with LLM Gateway" src="__img1" />

    You should see the response streaming from your chosen model. Check your [LLM Gateway dashboard](https://llmgateway.io/dashboard) to confirm the request appears in your usage logs.
  </Step>
</Steps>

## Adding More Models [#adding-more-models]

You can add any model from the [LLM Gateway models page](https://llmgateway.io/models) to your `models.json`. Just add entries to the `models` array:

```json
{
	"providers": {
		"llmgateway": {
			"baseUrl": "https://api.llmgateway.io/v1",
			"api": "openai-completions",
			"apiKey": "llmgtwy_your_api_key_here",
			"models": [
				{ "id": "gpt-5.5", "name": "GPT-5.5" },
				{ "id": "gpt-5.5-mini", "name": "GPT-5.5 Mini" },
				{ "id": "claude-opus-4-7", "name": "Claude Opus 4.7" },
				{ "id": "claude-sonnet-4-6", "name": "Claude Sonnet 4.6" },
				{ "id": "gemini-3.1-pro", "name": "Gemini 3.1 Pro" },
				{ "id": "gemini-3.1-flash", "name": "Gemini 3.1 Flash" },
				{ "id": "deepseek-v4", "name": "DeepSeek V4", "reasoning": true },
				{
					"id": "deepseek-v4-mini",
					"name": "DeepSeek V4 Mini",
					"reasoning": true
				}
			]
		}
	}
}
```

## Using Environment Variables for the API Key [#using-environment-variables-for-the-api-key]

Instead of hardcoding your key, you can reference an environment variable:

```json
{
	"providers": {
		"llmgateway": {
			"baseUrl": "https://api.llmgateway.io/v1",
			"api": "openai-completions",
			"apiKey": "LLM_GATEWAY_API_KEY",
			"models": [{ "id": "gpt-5.5", "name": "GPT-5.5" }]
		}
	}
}
```

Then set the variable in your shell profile:

```bash
export LLM_GATEWAY_API_KEY=llmgtwy_your_api_key_here
```

## Troubleshooting [#troubleshooting]

### Authentication Errors [#authentication-errors]

* Verify your API key is correct in `~/.pi/agent/models.json`
* Check that the base URL is set to `https://api.llmgateway.io/v1`
* Ensure your LLM Gateway account has sufficient credits

### Model Not Found [#model-not-found]

* Verify the model ID exists on the [models page](https://llmgateway.io/models)
* Model IDs are case-sensitive — copy them exactly as shown

### Connection Issues [#connection-issues]

* Check your internet connection
* Ensure `api` is set to `"openai-completions"` (not `"openai-responses"`)
* Monitor your usage in the LLM Gateway dashboard

<Callout type="info">
  Need help? Join our [Discord community](https://llmgateway.io/discord) for
  support and troubleshooting assistance.
</Callout>

## Benefits of Using LLM Gateway with Pi [#benefits-of-using-llm-gateway-with-pi]

* **Any Model**: Use GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V4, or 200+ others
* **Cost Tracking**: Every Pi request appears in your dashboard with token counts and costs
* **Caching**: Repeated requests hit cache automatically, saving money
* **One Key**: Manage all providers through a single API key
* **No Vendor Lock-in**: Switch models by changing one line in your config


# AWS Bedrock Integration
URL: https://docs.llmgateway.io/integrations/aws-bedrock


AWS Bedrock is Amazon's fully managed service that provides access to foundation models from leading AI companies. This guide shows how to create AWS Bedrock Long-Term API Keys and integrate them with LLM Gateway.

## Prerequisites [#prerequisites]

* An AWS account with Bedrock access enabled
* LLM Gateway account or self-hosted instance

## Overview [#overview]

AWS Bedrock supports **Long-Term API Keys** for simplified authentication. These keys provide direct API access without requiring IAM credentials or complex authentication flows.

## Create AWS Bedrock Long-Term API Key [#create-aws-bedrock-long-term-api-key]

<Steps>
  <Step>
    ### Enable Model Access in Bedrock [#enable-model-access-in-bedrock]

    1. Log into the **AWS Console**
    2. Navigate to **AWS Bedrock** service
    3. Go to **Model access** in the left sidebar
    4. Click **Manage model access**
    5. Enable the models you want to use (e.g., Claude 3.5, Llama 3)
    6. Wait for access to be granted (usually instant for most models)
  </Step>

  <Step>
    ### Create Long-Term API Key [#create-long-term-api-key]

    1. In AWS Bedrock console, navigate to **API Keys** in the left sidebar
    2. Click **Create Long-Term API Key**
    3. Set expiry date ("Never expires" is recommended)
    4. Click **Generate**
    5. **Important**: Copy the API key immediately - it's only shown once!
  </Step>
</Steps>

## Add to LLM Gateway [#add-to-llm-gateway]

<Steps>
  <Step>
    ### Navigate to Provider Keys [#navigate-to-provider-keys]

    1. Log into [LLM Gateway Dashboard](https://llmgateway.io/dashboard)
    2. Select your organization and project
    3. Go to **Provider Keys** in the sidebar
  </Step>

  <Step>
    ### Add AWS Bedrock Provider Key [#add-aws-bedrock-provider-key]

    1. Click **Add** for **AWS Bedrock**
    2. Paste your Long-Term API Key
    3. **Select Region Prefix** based on where you want to use your models:
       * **us.** - For US regions (`us-east-1`, `us-west-2`)
       * **eu.** - For European regions (`eu-central-1`, `eu-west-1`)
       * **global.** - For global/cross-region endpoints
    4. Click **Add Key**

    The system will validate your key and confirm the connection.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    Test your integration with a simple API call:

    ```bash
    curl -X POST https://api.llmgateway.io/v1/chat/completions \
      -H "Authorization: Bearer YOUR_LLMGATEWAY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "aws-bedrock/claude-3-5-sonnet",
        "messages": [
          {
            "role": "user",
            "content": "Hello from AWS Bedrock!"
          }
        ]
      }'
    ```

    Replace `YOUR_LLMGATEWAY_API_KEY` with your LLM Gateway API key.
  </Step>
</Steps>

## Available Models [#available-models]

Once configured, you can access all AWS Bedrock models through LLM Gateway:

* **Anthropic Claude**: `aws-bedrock/claude-3-5-sonnet`, `aws-bedrock/claude-3-5-haiku`
* **Meta Llama**: `aws-bedrock/llama-3-2-90b`, `aws-bedrock/llama-3-2-11b`
* **Amazon Titan**: `aws-bedrock/amazon.titan-text-express-v1`
* **And more...**

Browse all available models at [llmgateway.io/models](https://llmgateway.io/models?provider=aws-bedrock)

## Troubleshooting [#troubleshooting]

### "Model not available" error [#model-not-available-error]

* Verify you've enabled model access in AWS Bedrock console
* Check that the region where you created your key has access to the model
* Some models are only available in specific regions

### Rate limiting [#rate-limiting]

* AWS Bedrock has request quotas per model and region
* Monitor usage in AWS Bedrock console
* Consider requesting quota increases for high-volume workloads


# Azure Integration
URL: https://docs.llmgateway.io/integrations/azure


Azure provides access to OpenAI's powerful language models through Microsoft's enterprise cloud infrastructure. This guide shows how to create an Azure resource, deploy models, and integrate them with LLM Gateway.

<Callout type="info">
  Azure exposes three separate providers in LLM Gateway: `azure` for OpenAI
  models, `azure-ai-foundry` for third-party models served through the Azure
  Models inference endpoint, and `azure-anthropic` for Claude models on
  Microsoft Foundry. [Open an
  issue](https://github.com/theopenco/llmgateway/issues/new) to request support
  for other model types.
</Callout>

## Prerequisites [#prerequisites]

* An Azure account with an active subscription
* LLM Gateway account or self-hosted instance

## Overview [#overview]

Azure provides enterprise-grade access to OpenAI models with enhanced security, compliance, and regional availability. LLM Gateway integrates seamlessly with Azure deployments.

## Create Azure Resource [#create-azure-resource]

<Steps>
  <Step>
    ### Create an Azure OpenAI Resource [#create-an-azure-openai-resource]

    1. Log into the **Azure Portal** ([https://portal.azure.com](https://portal.azure.com))
    2. Click **Create a resource**
    3. Search for **Azure OpenAI** and select it
    4. Click **Create**
    5. Configure the resource:
       * **Subscription**: Select your Azure subscription
       * **Resource group**: Create new or select existing
       * **Region**: Choose a region (e.g., East US, West Europe)
       * **Name**: Enter a unique resource name (this will be your `<resource-name>`)
       * **Pricing tier**: Select Standard S0
    6. Click **Review + create**, then **Create**
    7. Wait for deployment to complete

    **Important**: Note your resource name - it will be used in the base URL: `https://<resource-name>.openai.azure.com`
  </Step>

  <Step>
    ### Deploy Models [#deploy-models]

    1. Navigate to your Azure resource in the Azure Portal
    2. Click **Go to Azure OpenAI Studio** or visit [https://oai.azure.com](https://oai.azure.com)
    3. In Azure Studio, select **Deployments** from the left sidebar
    4. Click **Create new deployment**
    5. Configure your deployment:
       * **Model**: Select a model (e.g., gpt-4o, gpt-4o-mini, gpt-4-turbo)
       * **Deployment name**: Enter a name (this must match the model identifier you'll use – use the pre-filled name)
       * **Model version**: Select the latest version
       * **Deployment type**: Global Standard
    6. Click **Create**
    7. Repeat for additional models you want to use

    **Note**: The deployment name must match the expected model name:

    * For `gpt-4o-mini` → deployment name should be `gpt-4o-mini`
    * For `gpt-35-turbo` → deployment name should be `gpt-35-turbo`
      etc.
  </Step>

  <Step>
    ### Get API Key [#get-api-key]

    1. In the Azure Portal, go to your Azure resource
    2. Click **Keys and Endpoint** in the left sidebar
    3. Copy **Key 1** or **Key 2**
    4. Note your **Endpoint** URL (should be `https://<resource-name>.openai.azure.com`)

    **Important**: Keep your API key secure - it provides access to your Azure deployments.
  </Step>
</Steps>

## Add to LLM Gateway [#add-to-llm-gateway]

<Steps>
  <Step>
    ### Navigate to Provider Keys [#navigate-to-provider-keys]

    1. Log into [LLM Gateway Dashboard](https://llmgateway.io/dashboard)
    2. Select your organization and project
    3. Go to **Provider Keys** in the sidebar
  </Step>

  <Step>
    ### Add Azure Provider Key [#add-azure-provider-key]

    1. Click **Add** for **Azure**
    2. Enter your **API Key** from Azure Portal
    3. Enter your **Resource Name** (the name from your Azure endpoint URL)
       * Example: If your endpoint is `https://my-openai-resource.openai.azure.com`, enter `my-openai-resource`
    4. Select your preferred **type** (Azure OpenAI or AI Foundry)
    5. Adapt the **Validation Model** to a model that you already deployed and is available
       This is a one time check to ensure the API key is valid and the model can be accessed.
    6. Click **Add Key**

    The system will validate your key and confirm the connection.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    Test your integration with a simple API call:

    ```bash
    curl -X POST https://api.llmgateway.io/v1/chat/completions \
      -H "Authorization: Bearer YOUR_LLMGATEWAY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "azure/gpt-4o-mini",
        "messages": [
          {
            "role": "user",
            "content": "Hello from Azure!"
          }
        ]
      }'
    ```

    Replace `YOUR_LLMGATEWAY_API_KEY` with your LLM Gateway API key.
  </Step>
</Steps>

## Available Models [#available-models]

Once configured, you can access your Azure deployments through LLM Gateway:

* **GPT-4o**: `azure/gpt-4o`
* **GPT-4o Mini**: `azure/gpt-4o-mini`
* **GPT-3.5 Turbo**: `azure/gpt-3.5-turbo` (note: use gpt-3.5-turbo as llmgateway model name instead of gpt-35-turbo)

**Note**: Only models you have deployed in Azure Studio will be available. Ensure your deployment names match the expected model identifiers.

Browse all available models at [llmgateway.io/models](https://llmgateway.io/models?provider=azure)

## Claude models on Microsoft Foundry [#claude-models-on-microsoft-foundry]

Claude models are served by Microsoft Foundry through Anthropic's Messages API rather than an OpenAI-compatible endpoint, so they use their own `azure-anthropic` provider in LLM Gateway.

1. In the Foundry portal, deploy the Claude model you want. The deployment name is what you pass as the model at inference time, so keep the default (the model ID) unless you have a reason to rename it.
2. Add an **Azure Anthropic** provider key in LLM Gateway with your Foundry API key and the resource name from `https://<resource-name>.services.ai.azure.com`.
3. Call it with the `azure-anthropic/<model>` model string:

```bash
curl -X POST https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer YOUR_LLMGATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "azure-anthropic/claude-opus-4-8",
    "messages": [
      {
        "role": "user",
        "content": "Hello from Microsoft Foundry!"
      }
    ]
  }'
```

If a request returns `DeploymentNotFound`, the model has not been deployed in that Foundry resource, or the deployment name does not match the model string you sent. Browse the Claude models routed through this provider at [llmgateway.io/models](https://llmgateway.io/models?provider=azure-anthropic).

## Troubleshooting [#troubleshooting]

### "Deployment not found" error [#deployment-not-found-error]

* Verify you've created a deployment in Azure Studio
* Ensure the deployment name exactly matches the model name you're requesting
* Check that the deployment is in the same resource as your API key

### "Resource not found" error [#resource-not-found-error]

* Verify the resource name is correct (check your Azure Portal endpoint URL)
* Ensure your API key belongs to the correct Azure resource
* Confirm the resource is in an active state in Azure Portal

### Rate limiting [#rate-limiting]

* Azure has Tokens Per Minute (TPM) quotas per deployment
* Monitor usage in Azure Studio under **Quotas**
* Request quota increases through Azure Portal if needed for high-volume workloads

### Region availability [#region-availability]

* Not all models are available in all Azure regions
* Check [Azure model availability](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#model-summary-table-and-region-availability) for your region
* Consider creating resources in multiple regions for better availability


# Vertex AI Anthropic Integration
URL: https://docs.llmgateway.io/integrations/vertex-anthropic


Run Claude models (Sonnet, Opus, Haiku) on Google Cloud Vertex AI through LLM Gateway. This guide shows how to set up a GCP service account and integrate it with LLM Gateway using automatic OAuth2 token management — no manual token rotation required.

## Prerequisites [#prerequisites]

* A Google Cloud project with billing enabled
* LLM Gateway account or self-hosted instance

## Set up Google Cloud [#set-up-google-cloud]

<Steps>
  <Step>
    ### Enable the Vertex AI API [#enable-the-vertex-ai-api]

    In the [Google Cloud Console](https://console.cloud.google.com/apis/library/aiplatform.googleapis.com), enable the **Vertex AI API** for your project.
  </Step>

  <Step>
    ### Enable Claude Models in Model Garden [#enable-claude-models-in-model-garden]

    Navigate to **Vertex AI > Model Garden** in the Cloud Console. Search for the Claude models you want to use and click **Enable** on each one.

    Available models:

    * `claude-sonnet-4-6`
    * `claude-sonnet-4-5`
    * `claude-haiku-4-5`
    * `claude-opus-4-5`
    * `claude-opus-4-6`
    * `claude-opus-4-7`
  </Step>

  <Step>
    ### Create a Service Account [#create-a-service-account]

    Create a service account with the required permissions:

    ```bash
    # Create the service account
    gcloud iam service-accounts create vertex-ai-caller \
      --display-name="Vertex AI Caller" \
      --project=YOUR_PROJECT_ID

    # Grant the Vertex AI User role
    gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
      --member="serviceAccount:vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
      --role="roles/aiplatform.user"
    ```
  </Step>

  <Step>
    ### Download the Service Account Key [#download-the-service-account-key]

    ```bash
    gcloud iam service-accounts keys create service-account.json \
      --iam-account=vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com
    ```

    Then convert it to a single-line string:

    ```bash
    cat service-account.json | tr -d '\n'
    ```

    Keep the output handy — you'll paste it into LLM Gateway in the next steps.
  </Step>
</Steps>

## Add to LLM Gateway [#add-to-llm-gateway]

<Steps>
  <Step>
    ### Navigate to Provider Keys [#navigate-to-provider-keys]

    1. Log into [LLM Gateway Dashboard](https://llmgateway.io/dashboard)
    2. Select your organization and project
    3. Go to **Provider Keys** in the sidebar
  </Step>

  <Step>
    ### Add Vertex Anthropic Provider Key [#add-vertex-anthropic-provider-key]

    1. Click **Add** for &#x2A;*Vertex AI (Anthropic)**
    2. Paste the single-line service account JSON as the **API Key**
    3. Leave **Region** empty to use the recommended `global` endpoint, or set a specific region (e.g. `us-east5`) if you need data residency
    4. Click **Add Key**

    The project ID is extracted automatically from the service account JSON — no separate project field is needed.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    ```bash
    curl -X POST https://api.llmgateway.io/v1/chat/completions \
      -H "Authorization: Bearer YOUR_LLMGATEWAY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "vertex-anthropic/claude-sonnet-4-6",
        "messages": [
          {
            "role": "user",
            "content": "Hello from Vertex Anthropic!"
          }
        ]
      }'
    ```

    Replace `YOUR_LLMGATEWAY_API_KEY` with your LLM Gateway API key.
  </Step>
</Steps>

## Self-Host Configuration [#self-host-configuration]

If you're self-hosting LLM Gateway, configure the provider via environment variables instead of the dashboard:

```bash
LLM_VERTEX_ANTHROPIC_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"YOUR_PROJECT_ID","private_key":"-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n","client_email":"vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com","token_uri":"https://oauth2.googleapis.com/token"}
LLM_VERTEX_ANTHROPIC_REGION=global
```

<Callout type="info">
  The project ID is extracted automatically from the service account JSON, so
  `LLM_VERTEX_ANTHROPIC_PROJECT` only needs setting to call a different project
  than the service account's own.
</Callout>

## How Token Refresh Works [#how-token-refresh-works]

LLM Gateway handles the OAuth2 token lifecycle automatically:

1. On first request, the service account JSON is parsed and used to sign a JWT
2. The JWT is exchanged for an OAuth2 access token via Google's token endpoint
3. The token is cached in Redis with a **50-minute TTL** (Google tokens expire after 60 minutes)
4. An in-memory cache avoids Redis round-trips on subsequent requests
5. When the cached token expires, a new one is generated transparently

This means:

* No manual `gcloud auth print-access-token` commands
* No cron jobs to refresh tokens
* Works at any request rate (token generation happens at most once per 50 minutes)
* Multi-instance deployments share the cached token via Redis

## Available Regions [#available-regions]

LLM Gateway defaults to the &#x2A;*`global`** endpoint, which Anthropic recommends: requests are routed dynamically to whichever region has capacity, and there is no pricing premium.

| Region            | Notes                                         |
| ----------------- | --------------------------------------------- |
| `global`          | Default — dynamic routing, no pricing premium |
| `us`              | Multi-region (US only); 10% premium           |
| `eu`              | Multi-region (EU only); 10% premium           |
| `us-east5`        | Columbus, Ohio; 10% premium                   |
| `us-central1`     | Iowa; 10% premium                             |
| `europe-west1`    | Belgium; 10% premium                          |
| `europe-west4`    | Netherlands; 10% premium                      |
| `asia-southeast1` | Singapore; 10% premium                        |

<Callout type="info">
  Regional and multi-region endpoints add a 10% pricing premium on Claude Sonnet
  4.5 and newer models. They are also required if you need single-region data
  residency or provisioned throughput. See [Anthropic's Vertex
  docs](https://platform.claude.com/docs/en/api/claude-on-vertex-ai#global-multi-region-and-regional-endpoints)
  for details.
</Callout>

## Available Models [#available-models]

Once configured, you can access Claude models on Vertex AI through LLM Gateway. Browse the full, up-to-date list at [llmgateway.io/models?provider=vertex-anthropic](https://llmgateway.io/models?provider=vertex-anthropic).

## Troubleshooting [#troubleshooting]

### 401 UNAUTHENTICATED / ACCESS\_TOKEN\_TYPE\_UNSUPPORTED [#401-unauthenticated--access_token_type_unsupported]

The gateway is sending an invalid token. Check:

* The service account JSON is valid and complete
* The service account has `roles/aiplatform.user` on the project

### 403 Permission Denied [#403-permission-denied]

The service account lacks permissions. Grant the `Vertex AI User` role:

```bash
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"
```

### Model Not Found [#model-not-found]

The Claude model may not be enabled in your project's Model Garden, or may not be available in the selected region. Check the [Model Garden](https://console.cloud.google.com/vertex-ai/model-garden) in Cloud Console.


# Activity
URL: https://docs.llmgateway.io/learn/activity


The Activity page shows a real-time log of every API request routed through LLM Gateway. Use it to debug requests, monitor performance, and track costs per call.

## Filters [#filters]

Filter the activity log using the controls at the top:

| Filter                      | Description                                             |
| --------------------------- | ------------------------------------------------------- |
| **Time range**              | Filter by a specific time period                        |
| **Unified reasons**         | Filter by completion reason (e.g., stop, length, error) |
| **Providers**               | Show requests for specific providers only               |
| **Models**                  | Show requests for specific models only                  |
| **Custom header key/value** | Filter by custom metadata headers attached to requests  |

## Activity List [#activity-list]

Each activity entry shows:

* **Status icon** — Green checkmark for completed, red circle for errors
* **Response preview** — First line of the model's response (when available)
* **Model** — The provider and model used (e.g., `google-vertex/gemini-3-pro-image`)
* **Cache status** — Whether the response was served from cache
* **Tokens** — Total tokens consumed (input + output)
* **Duration** — How long the request took
* **Cost** — Inference cost for the request
* **Source** — Where the request originated from
* **Discount** — Any discount applied (e.g., "20% off")
* **Status badge** — `completed`, `upstream_error`, `gateway_error`, etc.
* **Timestamp** — Relative time (e.g., "about 4 hours ago")

### Actions per Entry [#actions-per-entry]

* **Open in new tab** — View the full request detail in a new browser tab
* **Expand** — Expand inline to see more details

## Activity Detail [#activity-detail]

Click on any activity entry to view its full detail page.

### Summary Cards [#summary-cards]

Five cards at the top provide a quick overview:

| Card               | Description                     |
| ------------------ | ------------------------------- |
| **Duration**       | Total request time in seconds   |
| **Tokens**         | Total tokens consumed           |
| **Throughput**     | Tokens per second               |
| **Inference Cost** | Cost charged for this request   |
| **Cache**          | Whether the response was cached |

### Request Section [#request-section]

Details about the original request:

* **Requested Model** — The model ID sent in the API call
* **Used Model** — The actual model that served the request
* **Model Mapping** — The underlying model identifier
* **Provider** — The provider that handled the request
* **Requested Provider** — The provider specified in the request
* **Streamed** — Whether the response was streamed
* **Canceled** — Whether the request was canceled
* **Source** — The application or service that made the request

### Tokens Section [#tokens-section]

A detailed token breakdown:

* Prompt Tokens, Completion Tokens, Total Tokens
* Reasoning Tokens (for reasoning models)
* Image Input/Output Tokens (for vision/image models)
* Response Size

### Routing Section [#routing-section]

How LLM Gateway routed the request:

* **Selection** — The routing strategy used (e.g., `direct-provider-specified`)
* **Available** — Providers that were available for this model
* **Provider Scores** — Scoring breakdown showing availability, uptime, and latency for each provider

### Parameters Section [#parameters-section]

The model parameters sent with the request:

* Temperature, Max Tokens, Top P
* Frequency Penalty, Reasoning Effort
* Response Format


# Agents Monitoring
URL: https://docs.llmgateway.io/learn/agents


The Agents page groups your project's gateway traffic by the coding agent that
produced it — Claude Code, Cursor, OpenCode, Cline, Codex, DevPass Code and
others — so you can see which tool is spending your credits and drill into the
individual sessions behind the numbers.

<Callout type="info">
  Agents are identified by the `x-source` header their requests send. Anything
  that doesn't match a known agent stays out of this view — use
  [**Activity**](https://docs.llmgateway.io/learn/activity) for the full request log.
</Callout>

## Filtering the view [#filtering-the-view]

Two controls sit above the agent cards and both are stored in the URL, so a
filtered view can be bookmarked or shared:

| Control        | Options              | Default |
| -------------- | -------------------- | ------- |
| **Time range** | 1h, 4h, 24h, 7d, 30d | 7d      |
| **Usage mode** | All, Credits, BYOK   | All     |

**Credits** shows only traffic billed against your credit balance, **BYOK**
only traffic served by your own [provider keys](https://docs.llmgateway.io/learn/provider-keys). The
summary on the right of the row totals the agents, requests, and spend
currently in view.

## Agent Cards [#agent-cards]

Each agent that was active in the selected period gets a card:

| Field           | Description                                |
| --------------- | ------------------------------------------ |
| **Name**        | The agent, matched from its request source |
| **Total cost**  | Spend for this agent in the period         |
| **Requests**    | Number of API requests                     |
| **Tokens**      | Prompt and completion tokens combined      |
| **Last active** | When the agent last sent a request         |

Click a card to open its detail view.

## Agent Detail [#agent-detail]

The detail view splits the agent's requests into **sessions** — consecutive
requests with less than a 30-minute gap between them. Each session row shows
its time range, request count, total tokens, duration, and cost. Expand a
session to see the individual requests with their response previews, model,
cache status, tokens, cost, and source.

Sessions load as you scroll; **Load more sessions** fetches the next page
manually.

## Exporting to CSV [#exporting-to-csv]

**Export CSV** downloads every request in the selected period — not just the
sessions currently loaded — for the agent you are viewing. Use it to reconcile
agent spend against your own reporting.

For the DevPass equivalent of this page, see
[**Coding Agents**](https://docs.llmgateway.io/learn/coding-agents).


# Airside Fleet
URL: https://docs.llmgateway.io/learn/airside-fleet


The Fleet page is where a claimed carrier manages its model listings on
[Airside](https://docs.llmgateway.io/features/airside), the self-serve provider console. Each listing is
a "flight strip" showing its status, its currently approved pricing, and any
tariff filing still awaiting review.

## Listing statuses [#listing-statuses]

| Status                              | Meaning                                                                   |
| ----------------------------------- | ------------------------------------------------------------------------- |
| **Filed**                           | Drafted — waiting for the initial price filing to be approved             |
| **In service**                      | Approved and routable as `<provider>/<model>`; billed at the filed prices |
| **Rejected**                        | The initial filing was rejected; fix the price and file again             |
| **Delisted**                        | Retired — kept in history, no longer routable                             |
| **Awaiting clearance / Fare filed** | A filing (initial or price update) is pending review                      |

## Registering an aircraft [#registering-an-aircraft]

Click **Register aircraft**, pick the carrier (when your company holds more
than one), and fill in the model id, display name, context size, capabilities,
and the initial tariff. Prices are per token and accept exponent notation —
`2e-6` reads as $2 per million tokens.

<Callout type="info">
  New listings and every later price change are reviewed by the LLM Gateway team
  before taking effect. Everything else about a model — display name,
  description, context size, capability flags — applies immediately.
</Callout>

## Changing a price [#changing-a-price]

Pricing is never edited in place. **File fare** opens a tariff filing with the
new input/output prices; the model keeps serving at its current approved
pricing until the filing is approved. Only one filing per model can be pending
at a time, and every filing — with its review outcome and notes — is listed in
the console's filings history.

## Removing a model [#removing-a-model]

Deleting a drafted or rejected listing removes it outright. An in-service
model is **delisted** instead: it stops routing immediately but stays in your
history and analytics.

<Callout type="warning">
  Delisting takes effect within about a minute — in-flight requests finish, but
  new requests for the model are rejected.
</Callout>


# Analytics
URL: https://docs.llmgateway.io/learn/analytics


The Analytics page shows where a project's spend actually goes. It leads with the project's totals for the range, then breaks usage down — as a ranking and over time — so you can see what drives cost, requests, and tokens across any date range.

Open it from the **Analytics** item in the project sidebar. Like the rest of the dashboard, the page respects the shared date-range picker at the top, so every chart reflects the same window.

## Totals [#totals]

Three cards at the top show the project's **total cost**, **requests**, and **tokens** for the selected range, each with a sparkline of its daily values. They are computed from the same data as the charts below, so the total and the breakdown always agree.

## Choosing a breakdown [#choosing-a-breakdown]

The **Group by** selector switches which dimension the two charts break the project down by:

| Breakdown   | What it shows                                              |
| ----------- | ---------------------------------------------------------- |
| **Model**   | Spend per model (the default)                              |
| **API key** | Spend per API key in the project                           |
| **User**    | Spend per team member — Enterprise, owners and admins only |

**Breakdown by user** answers "what is each person on this team costing us?" in the same view as the project total. It is available to organization owners and admins on the Enterprise plan — see [Member Analytics](https://docs.llmgateway.io/learn/member-analytics).

Spend is attributed to whoever **created** the API key that made each request. Traffic from platform keys has no member behind it, so it is shown as **Unattributed** rather than dropped, keeping the per-member rows summing to the project total.

## Cost by Model [#cost-by-model]

A horizontal bar chart that ranks the models used in the selected range. Switch the metric with the tabs above the chart:

| Tab          | What it ranks                          |
| ------------ | -------------------------------------- |
| **Cost**     | Total spend per model, in USD          |
| **Requests** | Number of requests routed to the model |
| **Tokens**   | Total tokens (input + output)          |

Use it to spot the one or two models responsible for most of your bill, or to confirm that traffic is spread the way you expect.

## Cost by Model Over Time [#cost-by-model-over-time]

A stacked area chart that plots the same three metrics across the date range, with one band per model. It has the same **Cost / Requests / Tokens** tabs, plus a **Mappings / Canonical** toggle:

* **Mappings** — each model variant is shown separately, exactly as it was requested (for example a custom provider mapping is kept distinct from the built-in model).
* **Canonical** — variants of the same underlying model are collapsed into one canonical model (the provider prefix and tag are dropped), so `openai/gpt-5.5` and a custom mapping of it count as a single line.

Switch to **Canonical** when you care about the underlying model's total footprint; stay on **Mappings** when you need to compare specific routes.

The **Mappings / Canonical** toggle is specific to models — the API key and user breakdowns rank named entities, so there is nothing to collapse.

## How the data is computed [#how-the-data-is-computed]

Both charts derive from the same activity data the rest of the dashboard already reads — there is no separate analytics pipeline to wait on. Aggregation happens per time bucket and is timezone-correct, so totals line up with the Activity and Usage pages for the same range.

For a per-key view of the same breakdowns, see [API Keys](https://docs.llmgateway.io/learn/api-keys#per-key-statistics). For an org-wide, per-person view on the Enterprise plan, see [Member Analytics](https://docs.llmgateway.io/learn/member-analytics).


# API Keys
URL: https://docs.llmgateway.io/learn/api-keys


The API Keys page is the main place to create, secure, and operate the keys
your apps use to authenticate with LLM Gateway.

Use this page to:

* Create project-specific API keys
* Set all-time and recurring spend limits per key
* Set an expiration (TTL) so a key disables itself automatically
* Track usage for each key, including the active recurring window
* Rename keys and enable or disable them without deleting them
* Roll a key's secret in place if it may have leaked
* Configure IAM rules for model, provider, and pricing access

<Callout type="warning">
  API keys are shown in full only once, immediately after creation. Copy and
  store them securely before closing the dialog. New and rolled secrets are
  stored only as keyed HMAC-SHA-256 fingerprints; authentication fingerprints
  the presented secret and compares the result.
</Callout>

## Creating an API Key [#creating-an-api-key]

Click **Create API Key** and configure:

* **Name**: A label such as `production`, `staging`, or `ci`
* **Expiration (TTL)**: An optional time-to-live after which the key disables itself
* **All-time usage limit**: An optional lifetime spend cap for the key
* **Recurring usage limit**: An optional spend cap that resets on a schedule

Recurring limits support:

* Minimum window: **1 hour**
* Maximum window: **12 months**
* Units: **hour**, **day**, **week**, or **month**

This is useful when you want a key to stay below a fixed budget per hour, day,
week, or month, while still keeping a separate lifetime cap if needed.

## Expiration (TTL) [#expiration-ttl]

Turn on &#x2A;*Set expiration (TTL)** when creating a key to give it a limited
lifetime. Choose a value and a unit — **minutes**, **hours**, or **days** — and
the key is disabled automatically once that time passes. Leave it off for a key
that never expires.

Expired keys show an **Expired** indicator in the list and move to the
**Inactive** tab. To use one again, reactivate it and pick a **new future
expiration**:

* **Activate** an expired key and you'll be prompted to set a fresh TTL before it
  comes back online
* Keys with no TTL, or whose TTL is still in the future, can be enabled and
  disabled without setting a new expiration

This makes TTL keys ideal for temporary access — short-lived demos, CI runs, or
contractor keys that should not linger.

## Usage Limits [#usage-limits]

Each API key can enforce two independent limit types:

| Limit Type                | What it does                                                    |
| ------------------------- | --------------------------------------------------------------- |
| **All-time usage limit**  | Stops the key after it reaches a lifetime spend threshold       |
| **Recurring usage limit** | Stops the key after it reaches the budget for the active window |

Examples:

* `$50` all-time for a temporary integration key
* `$10 / 1 day` for a development key
* `$500 / 1 month` for a production service key

If a key hits either limit, requests using that key are rejected until the key
is updated or, for recurring limits, the next window begins.

### How recurring windows work [#how-recurring-windows-work]

Recurring usage is tracked separately from total lifetime usage.

* The dashboard shows the key's **Current Period** usage
* The active window also shows when it **resets**
* When the configured window expires, usage for that window resets automatically
* Updating the recurring limit configuration resets the current window and starts
  a new one

Usage includes both LLM Gateway credits and requests routed through your own
provider keys when applicable.

## API Keys List [#api-keys-list]

Each key in the list shows:

| Field              | Description                                                   |
| ------------------ | ------------------------------------------------------------- |
| **Name**           | The label you assigned to the key                             |
| **API Key**        | A masked preview of the key                                   |
| **Status**         | Whether the key is active or inactive, plus its expiry if set |
| **Created**        | When the key was created                                      |
| **Usage**          | Total tracked usage for the key                               |
| **Current Period** | Spend in the active recurring window, if configured           |
| **Limits**         | All-time and recurring limit summary                          |
| **IAM Rules**      | Whether model/provider/pricing access controls are configured |

Every configured limit renders as a gauge, so a key's headroom is visible without
opening it. A key is flagged **approaching** once usage reaches 80% of a limit and
**reached** once usage meets it — the all-time limit stays blocked until it is
raised or removed, while a recurring limit clears itself on its reset date.

Three filters sit above the list:

* **Status** — active (default), inactive, or all
* **Creator** — keys you created, or everyone's (hidden for developers, who only
  ever see their own)
* **Limits** — narrow to just the keys approaching or at a limit

A filter that would leave the list empty resets itself, so you never land on a
blank table.

## Actions [#actions]

The actions menu on each key offers:

* **View Statistics**: Open a dedicated analytics page for that key (see below)
* **Manage IAM Rules**: Restrict which models, providers, or pricing tiers the key can use
* **Rename Key**: Change the key's label without touching its secret or history
* **Activate / Deactivate Key**: Pause usage without deleting the key (reactivating
  an expired key prompts for a new expiration)
* **Roll Key**: Generate a new secret for the key and invalidate the old one (see below)
* **Update limits**: Change all-time or recurring limits
* **Delete**: Permanently remove the key

<Callout type="info">
  The Playground key appears with a **Managed** badge and only exposes its
  statistics. It cannot be renamed, rolled, deactivated, limited, assigned IAM
  rules, or deleted. Members with the developer role can only modify keys they
  created themselves.
</Callout>

## Rolling a Key [#rolling-a-key]

**Roll Key** replaces a key's secret while keeping the key itself. The name,
usage history and statistics, all-time and recurring limits (including the
current period window), IAM rules, and expiration all stay exactly as they were —
only the secret changes.

This is the fastest fix when a secret leaks into a commit, a log, or a shared
environment: you cut off the exposed value without recreating the key or losing
its spend tracking.

1. Open the key's actions menu and choose **Roll Key**
2. Confirm in the dialog
3. Copy the new secret and update every client using the old one

<Callout type="warning">
  The previous secret stops working immediately, and the new one is shown only
  once. Requests still using the old secret fail with `401 Unauthorized`.
</Callout>

## Per-Key Statistics [#per-key-statistics]

The **View Statistics** action opens a dedicated page scoped to a single API
key, so you can see exactly what that key is doing without filtering the whole
project.

The page respects the shared date-range picker and shows:

* **Summary cards** — the key's cost, tokens, requests, and error rate for the
  selected range.
* **Cost by Model** — a horizontal bar chart ranking the key's models by cost,
  requests, or tokens.
* **Cost by Model Over Time** — a stacked area chart of the same metrics, with a
  Mappings / Canonical toggle.

These are the same breakdowns as the project [Analytics](https://docs.llmgateway.io/learn/analytics) page,
narrowed to the one key — useful for confirming a key is healthy and spending on
the models you expect.

## IAM Rules [#iam-rules]

IAM rules let you narrow what an API key is allowed to access.

Supported rule types include:

* **Allow/Deny models**
* **Allow/Deny providers**
* **Allow/Deny pricing**
* **Allow/Deny IP ranges (CIDR)** — Enterprise plan only

Use IAM rules when you want a key to be valid, but only for a specific subset of
models or providers. For a deeper explanation, see the [API Keys & IAM Rules
feature page](https://docs.llmgateway.io/features/api-keys).

Org admins can additionally set [member-level IAM
rules](https://docs.llmgateway.io/features/api-keys#member-level-iam-rules) on the [Team
page](https://docs.llmgateway.io/learn/team). Those act as a ceiling for every key the member creates:
key rules can only narrow access within them, never expand it. The IAM page
shows a notice when organization-level restrictions apply to your keys.

## Plan Limits [#plan-limits]

The page also shows how many API keys you are using relative to your plan
allowance. The cap counts **active** keys across every project in the
organization, so deactivated and deleted keys do not count against it.

* **Free**: Standard API key count limit
* **Pro**: A larger allowance
* **Enterprise**: Custom limits

Owners and admins can also set a per-member key cap on the [Team
page](https://docs.llmgateway.io/learn/team), which applies on top of the organization-wide allowance.

If you reach the limit, the **Create API Key** button is disabled until you
delete or deactivate unused keys, or upgrade.


# Audit Logs
URL: https://docs.llmgateway.io/learn/audit-logs


The Audit Logs page provides a complete history of all actions performed within your organization, essential for compliance and security monitoring.

<Callout type="info">
  Audit Logs are available on the [**Enterprise
  plan**](https://llmgateway.io/enterprise). Owner or Admin role is required.
</Callout>

## Filters [#filters]

Narrow down the log entries:

* **Action** — Filter by action type (create, delete, update, etc.)
* **Resource type** — Filter by resource (API, IAM, API Keys, etc.)

Both filters are populated dynamically based on the actions recorded in your organization.

## Audit Log Entries [#audit-log-entries]

Each log entry shows:

| Field             | Description                                                  |
| ----------------- | ------------------------------------------------------------ |
| **Timestamp**     | Exact time of the action (formatted as MMM d, yyyy HH:mm:ss) |
| **User**          | Name and email of the person who performed the action        |
| **Action**        | What was done (e.g., "API Keys → create")                    |
| **Resource type** | The type of resource affected (shown as a badge)             |
| **Resource ID**   | Identifier of the affected resource (with copy button)       |
| **Details**       | Additional metadata about the action                         |

## Pagination [#pagination]

The log supports infinite scrolling with a **Load More** button to view older entries. Entries are sorted newest first.


# Billing
URL: https://docs.llmgateway.io/learn/billing


The Billing page is your central hub for managing credits, plans, and payment methods.

## Credits [#credits]

Displays your current credit balance. Credits are consumed as you make API requests through the gateway. Click **Top Up Credits** to add more credits to your account.

## Fees [#fees]

Top-ups are charged the credit amount plus the following fees:

* **Platform fee** — A flat 5% fee applied to every credit purchase.
* **International card fee** — An additional 1.5% fee applied when paying with a non-US issued card. This covers the higher processing cost charged by the card network for international transactions. Cards issued in the United States are not subject to this fee.

The full breakdown (credits, platform fee, and — when applicable — the international card fee) is shown in the top-up dialog before you confirm payment, so the total charge is always transparent.

## Plan Management [#plan-management]

View and manage your subscription:

* See your current plan (Free or Enterprise)
* Billing cycle information
* Click **Manage Subscription** to upgrade, downgrade, or cancel

## Payment Methods [#payment-methods]

Manage your saved payment methods:

* Add a new credit card or payment method
* View existing payment methods
* Update billing information

## Auto Top-up Settings [#auto-top-up-settings]

Configure automatic credit top-ups so you never run out:

* **Enable/disable** auto top-up
* **Threshold** — The credit balance that triggers a top-up
* **Amount** — How many credits to add when the threshold is reached

This ensures uninterrupted service by automatically replenishing your credits when they run low.


# Lounge Memberships
URL: https://docs.llmgateway.io/learn/chat-plans


Lounge memberships (formerly Chat Plans) are optional monthly subscriptions for [Lounge](https://lounge.llmgateway.io), the LLM Gateway chat app. Instead of paying per request from your pay-as-you-go balance, a membership gives you a pool of monthly credits worth more than you pay — so heavy chat usage costs less.

## Plans [#plans]

There are three tiers, billed monthly:

| Plan        | Price  | Monthly value    | Models                                                                       |
| ----------- | ------ | ---------------- | ---------------------------------------------------------------------------- |
| **Starter** | $9/mo  | \~2× the value   | Most chat models — Claude Haiku & Sonnet, GPT-5-mini, Gemini Flash, and more |
| **Plus**    | $19/mo | \~2.5× the value | Everything in Starter **plus** frontier models                               |
| **Pro**     | $49/mo | \~3× the value   | All models, highest monthly allowance                                        |

The credit multiplier is tapered: the larger the plan, the more usage value each dollar buys at provider rates.

<Callout type="info">
  **Frontier models** — flagship models such as Claude Opus, GPT-5, Gemini 2.5
  Pro, and Grok 4 are included on **Plus** and **Pro**. The Starter plan covers
  the broad catalog of everyday chat models but does not include these frontier
  models.
</Callout>

## How credits work [#how-credits-work]

* **Monthly reset** — Your plan credits refresh at the start of each billing cycle. Unused credits do **not** roll over to the next month.
* **Plan credits drain first** — Requests made from the chat app draw down your plan's monthly credits before anything else.
* **Pay-as-you-go fallback** — Once your monthly credits are used up, the chat app falls back to your regular pay-as-you-go balance, which never expires. You can keep chatting without interruption.

## Managing your plan [#managing-your-plan]

* Open the **Pricing** page from the Lounge sidebar to compare tiers and subscribe.
* Your active membership appears in the Lounge sidebar with a badge, alongside how many credits remain for the cycle.
* **Upgrading** takes effect immediately: you're charged the new tier's full price, your billing cycle restarts, and your credits reset to the new tier's full monthly allowance. Unspent credits from the old cycle don't carry over.
* **Downgrading** also takes effect immediately: you move to the lower tier's allowance right away, and the unused portion of what you already paid for the higher tier is credited against your next invoice.
* **Cancelling** takes effect at the end of the period you've already paid for — you keep access until then.


# Coding Agents
URL: https://docs.llmgateway.io/learn/coding-agents


The Coding Agents section of the [DevPass dashboard](https://devpass.llmgateway.io/dashboard)
breaks your DevPass usage down by the tool that produced it, so you can tell at
a glance whether your allowance is going to Claude Code, Cursor, OpenCode, or
something else.

Agents are recognised from the `x-source` header their requests send. Supported
tools include DevPass Code, Claude Code, OpenCode, Cursor, Autohand Code,
Empryo, SoulForge, Cline, Codex CLI, GitHub Copilot, n8n and OpenClaw — each
one has a setup guide under [Guides](https://docs.llmgateway.io/guides).

<Callout type="info">
  The overview lists the **last 7 days**. Open an agent to pick a wider window.
  If nothing shows up yet, point your tool at DevPass with the two environment
  variables from the dashboard's Quick start card and run it once.
</Callout>

## Agent Cards [#agent-cards]

| Field           | Description                                      |
| --------------- | ------------------------------------------------ |
| **Name**        | The coding tool, matched from its request source |
| **Total cost**  | Spend attributed to this agent                   |
| **Requests**    | Number of API requests                           |
| **Tokens**      | Prompt and completion tokens combined            |
| **Last active** | When the agent last sent a request               |

Click a card to open the agent's detail page.

## Agent Detail [#agent-detail]

The header repeats the agent's totals for the selected period and holds the
two controls for the page:

* **Time range** — 1h, 4h, 24h, 7d or 30d (7d by default), stored in the URL so
  the view can be shared.
* **Export CSV** — downloads every request in the period, not just the rows
  currently loaded.

Below it are three blocks:

**Model usage chart** — spend over time, split by model.

**Model usage table** — one row per model with Provider, Requests, Tokens,
Cost, Cache cost, and a share-of-tokens bar. Every column header sorts.

**Requests** — the individual calls, newest first, each showing the model used,
cache status, token count and cost. More rows load as you scroll, or with
**Load more**.

## Related pages [#related-pages]

* [**Model Categories & Fair Use**](https://docs.llmgateway.io/learn/model-categories) — which models
  count against the premium allowance
* [**Agents Monitoring**](https://docs.llmgateway.io/learn/agents) — the same breakdown for a regular
  gateway project, with sessions and BYOK filtering


# Dashboard
URL: https://docs.llmgateway.io/learn/dashboard


The Dashboard is the first page you see after logging in. It provides a high-level overview of your project's LLM usage, costs, and performance at a glance.

## Date Range [#date-range]

At the top of the page, you can toggle the date range for all dashboard metrics:

* **7 days** — Last 7 days of data (default)
* **30 days** — Last 30 days of data
* **Custom** — Pick a custom start and end date

### Display time zone [#display-time-zone]

Timestamps and chart buckets follow the display time zone set under **Settings →
Account**: either your local zone or UTC. The preference drives both how a
datetime renders and the bucketing the analytics endpoints do, so a chart's bars
and its axis labels always agree. It is stored in a browser cookie rather than
your profile, so the server renders the correct zone on first paint, and the
DevPass dashboard reads the same preference from its own **Settings** page.

## Stat Cards [#stat-cards]

The dashboard displays eight metric cards in two rows:

### Top Row [#top-row]

| Card                     | Description                                                              |
| ------------------------ | ------------------------------------------------------------------------ |
| **Organization Credits** | Your current available credit balance                                    |
| **Total Requests**       | Number of API requests in the selected period, with cache hit percentage |
| **Total Cost**           | Total inference cost for the period, including storage costs             |
| **Total Savings**        | Savings from discounts during the selected period                        |

### Bottom Row [#bottom-row]

| Card                     | Description                                                         |
| ------------------------ | ------------------------------------------------------------------- |
| **Input Tokens & Cost**  | Total prompt tokens sent and their associated cost                  |
| **Output Tokens & Cost** | Total completion tokens received and their associated cost          |
| **Cached Tokens & Cost** | Tokens served from cache (if caching is enabled) and the cost saved |
| **Most Used Model**      | The model with the highest request count, along with its provider   |

## Usage Overview Chart [#usage-overview-chart]

Below the stat cards, a chart visualizes your usage over time. You can toggle between two views using the dropdown:

* **Costs** — Shows input, output, and cached input costs as a stacked area chart
* **Requests** — Shows request volume over time

The chart is filtered by the currently selected project.

## Quick Actions [#quick-actions]

A sidebar panel provides shortcuts to common tasks:

* **Manage API Keys** — Go to the API Keys page
* **Provider Keys** — Configure your own provider keys
* **View Activity** — See detailed request logs
* **Usage & Metrics** — Dive into usage analytics
* **Model Usage** — View per-model usage breakdown

## Cost Breakdown [#cost-breakdown]

A donut chart showing how your costs are distributed across different models and providers. Each segment is color-coded and labeled with the model name and cost, making it easy to identify your biggest cost drivers.

## Errors & Reliability [#errors--reliability]

Displays two key reliability metrics:

* **Error Rate** — Percentage of failed requests over the selected period
* **Uptime** — Gateway availability percentage

## Recent Activity [#recent-activity]

A table showing your most recent API requests with key details like model, status, tokens, duration, and cost. Click any entry to view the full request detail.

## Header Actions [#header-actions]

Two buttons in the top-right corner:

* **Create API Key** — Quickly create a new API key for your project
* **Top Up Credits** — Add credits to your organization balance


# Dynamic Routes
URL: https://docs.llmgateway.io/learn/dynamic-routes


The Dynamic Routes page in your project settings lets you define named routing flows — conditions on the request, percentage-based A/B splits, and model targets with provider fallback — and invoke them by putting `dynamic/<name>` in the `model` field instead of a concrete model. Routing logic moves out of your application code: change the flow in the dashboard and every client picks it up on the next request, no deploy needed.

<Callout type="info">
  Dynamic routes are available on the [**Enterprise
  plan**](https://llmgateway.io/enterprise) and can be managed by organization
  owners and admins.
</Callout>

## Creating a Route [#creating-a-route]

Enter a name under **New route** and click **Create**. Route names are lowercase slugs (letters, digits, and single hyphens) and become part of the model string — a route named `support` is invoked as `dynamic/support`. New routes start with a starter graph you can edit right away.

The route list shows each route's state at a glance: the published version badge (for example **v2**), **unpublished** when no version has been published yet, and **off** when the route is disabled.

## The Visual Editor [#the-visual-editor]

The **Visual** tab is a drag-and-drop canvas. Add nodes from the palette (drag onto the canvas or click), wire branches by connecting the round handles, and select any node to edit its settings in the inspector — conditions, split weights, the target model (with a searchable catalog picker), and an ordered provider fallback list.

Every flow starts at the **Start** node and ends at either a **Model** node (route the request) or an **End** node (reject the request with a `400`):

| Node type       | What it does                                                                                           |
| --------------- | ------------------------------------------------------------------------------------------------------ |
| **Model**       | Routes to a catalog model; an optional provider list restricts routing and sets the fallback order     |
| **Conditional** | If/else branching on request headers, body fields, or request metadata — first matching condition wins |
| **Percentage**  | Weighted traffic splits, deterministic per session so a conversation keeps its A/B assignment          |
| **End**         | Explicitly rejects the request with a `400`                                                            |

See the [dynamic routes feature docs](https://docs.llmgateway.io/features/dynamic-routes) for the full condition operator and field reference.

## The JSON Editor [#the-json-editor]

The **JSON** tab edits the same graph as raw JSON — useful for copy-pasting flows between projects or generating them programmatically. Both tabs stay in sync, and both validate against the full schema before saving: unknown models, unwired branches, cycles, and invalid conditions are listed under **Fix before saving** and block the save until resolved, so an invalid graph can never serve traffic.

## Publishing and Rolling Back [#publishing-and-rolling-back]

Edits go to the route's draft; live traffic is never affected until you click **Publish**, which snapshots the draft as an immutable, numbered version and re-validates it against the current model catalog. The **Versions** card lists the full history — **Load into draft** copies an old version into the editor, and **Publish this** rolls back to it instantly.

A route serves requests only while it is **Enabled** and has a published version; otherwise requests using `dynamic/<name>` are rejected with a `404`.

## Invoking a Route [#invoking-a-route]

Use the route name in place of a model on any OpenAI-compatible endpoint:

```bash
curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "x-user-tier: paid" \
  -d '{
    "model": "dynamic/support",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

Once the flow resolves to a model, the request goes through the same weighted [smart routing](https://docs.llmgateway.io/features/routing) as any other — provider scoring, sticky sessions, and automatic cross-provider fallback included. Each request's [activity log](https://docs.llmgateway.io/learn/activity) records the route name, version, and the node path the evaluation took.


# Guardrails
URL: https://docs.llmgateway.io/learn/guardrails


The Guardrails page lets you configure content safety rules that automatically scan and filter API requests before they reach the LLM provider.

<Callout type="info">
  Guardrails are available on the [**Enterprise
  plan**](https://llmgateway.io/enterprise). Owner or Admin role is required.
</Callout>

## Main Toggle [#main-toggle]

A global toggle at the top enables or disables all guardrails for your organization. Click **Save Changes** to apply.

## Organization vs. Project [#organization-vs-project]

The same page exists in each project's settings. By default a project uses the organization guardrails, and the project page shows them read-only alongside a banner naming the project — for example "Enterprise Project uses the organization guardrails".

Turn off **Use organization guardrails** on the project page to give that project its own configuration and custom rules, which then replace the organization's for that project's traffic. The project starts from a copy of the organization's current settings, so opting out does not drop the rules that were already in force — edit down from there and click **Save Changes**.

The organization page names any projects that override it ("The settings below apply to every project except …"), so it is always clear where these settings do and do not apply.

## System Rules [#system-rules]

Six built-in rules with individual enable/disable toggles:

| Rule                            | Description                                                          |
| ------------------------------- | -------------------------------------------------------------------- |
| **Prompt Injection Detection**  | Detects attempts to override or manipulate system instructions       |
| **Jailbreak Prevention**        | Identifies attempts to bypass safety measures                        |
| **PII Detection**               | Identifies personal information like emails, phone numbers, and SSNs |
| **Secrets Detection**           | Detects API keys, passwords, and credentials                         |
| **File Type Restrictions**      | Controls which file types can be uploaded                            |
| **Document Leakage Prevention** | Detects attempts to extract confidential documents                   |

Each rule has an action dropdown to configure the response:

* **Block** — Reject the request entirely
* **Redact** — Remove or mask sensitive content, then continue
* **Warn** — Log the violation but allow the request

## File Restrictions [#file-restrictions]

Configure file upload limits:

* **Max file size** — Set the maximum file size in MB
* **Allowed file types** — Add or remove permitted MIME types

## Custom Rules [#custom-rules]

Create organization-specific rules by clicking **Add Rule**:

* **Blocked Terms** — Block specific words or phrases
* **Custom Regex** — Match patterns with regular expressions
* **Topic Restriction** — Restrict content related to specific topics

Each custom rule can be individually enabled/disabled or deleted.

Learn more about guardrails in the [Guardrails feature docs](https://docs.llmgateway.io/features/guardrails).


# Introduction
URL: https://docs.llmgateway.io/learn


The LLM Gateway dashboard gives you full control over your LLM API usage, costs, and configuration. This section walks you through every page in the platform, grouped by product.

## AI Gateway [#ai-gateway]

Configure how requests flow through the gateway — keys, safety rules, and programmatic provisioning:

* [**API Keys**](https://docs.llmgateway.io/learn/api-keys) — Create and manage your API keys, plus per-key statistics
* [**Provider Keys**](https://docs.llmgateway.io/learn/provider-keys) — Bring your own provider API keys
* [**Models**](https://docs.llmgateway.io/learn/models) — Browse catalog and custom models with compliance eligibility
* [**Structured Outputs**](https://docs.llmgateway.io/learn/structured-outputs) — Soft JSON output versus strict JSON schema enforcement
* [**Preferences**](https://docs.llmgateway.io/learn/preferences) — Project-level settings like caching and mode
* [**Guardrails**](https://docs.llmgateway.io/learn/guardrails) — Content safety rules and filters
* [**Security Events**](https://docs.llmgateway.io/learn/security-events) — Monitor guardrail violations
* [**Payments SDK**](https://docs.llmgateway.io/learn/sdk-settings) — Embed end-user payments and credit purchases into your own app
* [**Dynamic Routes**](https://docs.llmgateway.io/learn/dynamic-routes) — Named, versioned routing flows with conditions and A/B splits (Enterprise)
* [**Master Keys**](https://docs.llmgateway.io/learn/master-keys) — Provision projects and API keys programmatically (Enterprise)

## Observability [#observability]

Monitor usage, spend, errors, and performance across every request:

* [**Dashboard**](https://docs.llmgateway.io/learn/dashboard) — Overview of your usage, costs, and performance
* [**Activity**](https://docs.llmgateway.io/learn/activity) — Detailed logs of every API request
* [**Agents Monitoring**](https://docs.llmgateway.io/learn/agents) — Cost, tokens, and sessions per coding agent
* [**Model Usage**](https://docs.llmgateway.io/learn/model-usage) — Usage breakdown by model
* [**Usage & Metrics**](https://docs.llmgateway.io/learn/usage-metrics) — Requests, errors, cache rates, and cost trends
* [**Analytics**](https://docs.llmgateway.io/learn/analytics) — Cost, requests, and tokens broken down by model
* [**Organization Analytics**](https://docs.llmgateway.io/learn/org-analytics) — Cost and usage rolled up across every project
* [**Member Analytics**](https://docs.llmgateway.io/learn/member-analytics) — Per-member cost and usage breakdowns (Enterprise)
* [**Audit Logs**](https://docs.llmgateway.io/learn/audit-logs) — Complete history of organization actions

## DevPass [#devpass]

Flat-price dev plans for AI coding tools:

* [**Coding Agents**](https://docs.llmgateway.io/learn/coding-agents) — See which coding tools your DevPass key is used with
* [**Model Categories & Fair Use**](https://docs.llmgateway.io/learn/model-categories) — How models are categorized and premium fair-use caps
* [**Reset Passes**](https://docs.llmgateway.io/learn/reset-passes) — Instantly restore the DevPass weekly premium allowance
* [**Pay-As-You-Go Overflow**](https://docs.llmgateway.io/learn/payg-overflow) — Keep coding past the monthly allowance with an opt-in credits balance

## Lounge [#lounge]

The members' lounge for AI — chat, media studios, and memberships:

* [**Lounge Chat**](https://docs.llmgateway.io/learn/playground) — Chat with 200+ models in one interface
* [**Group Chat**](https://docs.llmgateway.io/learn/playground-group) — Watch multiple models discuss and collaborate on your prompt
* [**Image Studio**](https://docs.llmgateway.io/learn/playground-image) — Generate images using AI models
* [**Video Studio**](https://docs.llmgateway.io/learn/playground-video) — Generate videos using AI models
* [**Audio Studio**](https://docs.llmgateway.io/learn/playground-audio) — Generate speech from text using AI voices
* [**Voice Calls**](https://docs.llmgateway.io/learn/playground-realtime) — Have a live speech-to-speech conversation with a model
* [**Lounge Memberships**](https://docs.llmgateway.io/learn/chat-plans) — Monthly membership plans for the Lounge chat app

## Airside [#airside]

The self-serve console for model providers ("carriers"):

* [**Airside Fleet**](https://docs.llmgateway.io/learn/airside-fleet) — List models, file tariffs, and manage your carrier's fleet

## Account & Billing [#account--billing]

Manage your organization, team, and payments:

* [**Billing**](https://docs.llmgateway.io/learn/billing) — Credits, plans, and payment methods
* [**Transactions**](https://docs.llmgateway.io/learn/transactions) — Payment and credit history
* [**Invoices**](https://docs.llmgateway.io/learn/invoices) — Download invoices and credit notes
* [**Referrals**](https://docs.llmgateway.io/learn/referrals) — Earn credits by referring others
* [**Policies**](https://docs.llmgateway.io/learn/policies) — Data retention configuration
* [**Org Preferences**](https://docs.llmgateway.io/learn/org-preferences) — Organization name and billing details
* [**Team**](https://docs.llmgateway.io/learn/team) — Manage members, roles, and shared developer policy


# Invoices
URL: https://docs.llmgateway.io/learn/invoices


Every completed payment gets a PDF invoice, and every refund a credit note. The
invoice is emailed to your billing address when the payment succeeds, and both
can be downloaded at any time from [**Transactions**](https://docs.llmgateway.io/learn/transactions) in
your organization settings.

## Downloading an invoice [#downloading-an-invoice]

1. Open **Settings → Transactions** in the dashboard.
2. Find the payment in the list.
3. Click **Invoice** on that row — the PDF downloads as
   `invoice-<transaction-id>.pdf`.

Refunds produce a **Credit note** instead of an invoice, downloaded the same
way. It shows the original purchase amount, the percentage refunded, and the
refunded total as a negative line.

<Callout type="info">
  Only **completed** payments with a positive amount have a document. Plan
  cancellations, plan endings, and free credit gifts have nothing to invoice, so
  those rows have no download button.
</Callout>

The invoice number is the transaction ID, so it matches the row you downloaded
it from.

## Putting company details on the invoice [#putting-company-details-on-the-invoice]

Company name, address, tax ID and invoice notes come from your organization's
billing information, under **Settings → Billing**:

| Field                   | Appears on the invoice as                      |
| ----------------------- | ---------------------------------------------- |
| **Email Address**       | The billing email address invoices are sent to |
| **Company Name**        | The billed company                             |
| **Billing Address**     | The billing address block                      |
| **Tax ID / VAT Number** | Your tax identification number                 |
| **Invoice Notes**       | A free-text footer (e.g. a PO number)          |

<Callout type="warning">
  Fill these in **before** you pay. The emailed invoice is generated at payment
  time and keeps the details as they were then, while a download from
  Transactions is rendered fresh from your current details — so the two copies
  of one invoice number can differ after you edit this page. Treat the emailed
  PDF as the record of what was invoiced, and for a lasting change get the
  details right before the next payment.
</Callout>

## DevPass invoices [#devpass-invoices]

DevPass subscriptions are billed on their own workspace. Open
**Billing** in the [DevPass dashboard](https://devpass.llmgateway.io/dashboard)
and use the **Billing history** table — every charge, refund and plan change is
listed there with its own download button in the **Document** column.

## Related pages [#related-pages]

* [**Transactions**](https://docs.llmgateway.io/learn/transactions) — the full payment and credit history
* [**Billing**](https://docs.llmgateway.io/learn/billing) — credits, plans, and payment methods


# Master Keys
URL: https://docs.llmgateway.io/learn/master-keys


The Master Keys page lets you create and manage org-scoped bearer tokens for the `/v1/master/*` API — so your own backend can provision projects, gateway API keys, and IAM rules programmatically, without going through the dashboard.

<Callout type="info">
  Master Keys are available on the [**Enterprise
  plan**](https://llmgateway.io/enterprise). Contact us at [contact@llmgateway.io](mailto:contact@llmgateway.io)
  to enable them for your organization.
</Callout>

## Creating a Master Key [#creating-a-master-key]

Click **Create Master Key**, give the key a descriptive name (for example, "Provisioning backend"), and click create.

<Callout type="warning">
  The plain token (prefixed `llmgmk_`) is shown **only once** at creation time.
  Copy it immediately and store it securely — only an HMAC-SHA256 hash is kept
  in the database.
</Callout>

## Key List [#key-list]

Each master key in the list shows:

| Field          | Description                                            |
| -------------- | ------------------------------------------------------ |
| **Name**       | The description you gave the key                       |
| **Master Key** | Masked form of the token, for identification           |
| **Status**     | Active or inactive                                     |
| **Created**    | When the key was created                               |
| **Created By** | The team member who created it                         |
| **Last Used**  | Timestamp of the key's most recent `/v1/master/*` call |

## Managing Keys [#managing-keys]

Use the actions menu on each row to:

* **Activate / Deactivate** — Temporarily suspend a key without deleting it. Inactive keys receive a 401 on every request.
* **Delete** — Permanently revoke the key. Any system using it immediately loses access to the `/v1/master/*` API.

Every create, delete, and status change is recorded in your organization's [audit log](https://docs.llmgateway.io/learn/audit-logs).

## Limits [#limits]

Each organization can have up to **10 active master keys**. A usage bar above the list shows how many you've used; contact us if you need more.

## Using a Master Key [#using-a-master-key]

Master keys authenticate against the programmatic provisioning API:

```bash
curl https://internal.llmgateway.io/v1/master/projects \
  -H "Authorization: Bearer llmgmk_..."
```

See the [Master Keys feature documentation](https://docs.llmgateway.io/features/master-keys) for the full endpoint reference, including project management, gateway API key provisioning, per-key IAM rules, and [member-level IAM rules](https://docs.llmgateway.io/features/master-keys#member-iam-rules) (addressable by membership id or email).


# Member Analytics
URL: https://docs.llmgateway.io/learn/member-analytics


Member Analytics breaks your organization's usage down by person, so you can see who is spending what across every project. It lives on the **Team** page and is available on the **Enterprise** plan.

<Callout type="info">
  Member analytics require the **Enterprise plan** and an organization **owner**
  or **admin** role. Non-enterprise organizations see an upgrade card, and
  members without admin access see an access notice instead of the data.
</Callout>

## Members Table [#members-table]

The Team page adds a usage table sorted by spend, so the heaviest users surface first. Each row shows that member's totals for the selected date range:

| Column         | Description                                  |
| -------------- | -------------------------------------------- |
| **Member**     | Name and email of the team member            |
| **Cost**       | Total spend attributed to the member, in USD |
| **Tokens**     | Total tokens (input + output)                |
| **Requests**   | Number of requests                           |
| **Error rate** | Share of the member's requests that failed   |
| **API keys**   | How many API keys the member created         |

Usage is attributed by **who created each API key** — spend lands on the member who owns the key that made the request, which is the only link between usage and a user.

## Member Detail [#member-detail]

Click a member to open their detail page, scoped to the same date range:

The detail view includes:

* **Summary cards** — the member's cost, tokens, requests, and error rate for the range.
* **Most used** — their top model, provider, and app.
* **Cost by model** — the same breakdown as the project [Analytics](https://docs.llmgateway.io/learn/analytics) page, scoped to this member.
* **Top providers and top apps** — tables ranking where the member's traffic goes.

This makes it easy to attribute cost to a team, investigate a spike, or confirm a member is using the models and providers you expect.


# Model Categories & Fair Use
URL: https://docs.llmgateway.io/learn/model-categories


Every model in the gateway is sorted into a category. Categories power dashboard filtering, analytics, and — for DevPass coding plans — the fair-use limits that keep flagship models available to everyone.

## Categories [#categories]

| Category     | Description                                                                                                                  |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| **Premium**  | High-cost frontier / flagship models — priced at **$15+ per million output tokens*&#x2A; or **$5+ per million input tokens** |
| **Standard** | Every other model — the broad catalog of fast, cost-effective everyday models                                                |

The classification is computed automatically from catalogue prices — there is no hand-picked list. To see exactly which models are premium right now, check the live [**Premium Models**](https://llmgateway.io/models/premium) page, or browse the full catalog on the [**Supported Models**](https://llmgateway.io/models) page and filter by use case, capabilities, provider, price, and context size. On the [DevPass models directory](https://devpass.llmgateway.io/models), you can additionally filter by pricing tier, since the premium/standard distinction only affects DevPass fair-use limits.

## Fair-use caps on premium models (DevPass only) [#fair-use-caps-on-premium-models-devpass-only]

<Callout type="warn">
  Fair-use caps apply **only to DevPass** — the fixed-price monthly plans for
  coding tools (Lite, Pro, Max). They do **not** apply to the LLM Gateway API or
  pay-as-you-go credits: when you call the API directly, premium models are
  limited only by your credit balance, with no weekly cap.
</Callout>

Premium models are the most expensive to run, so DevPass plans apply a **weekly fair-use cap** on premium usage. The cap works on a fixed 7-day window: the window opens with your first premium-model request, all premium usage during those 7 days counts against the cap, and the full allowance resets at once when the window ends. It sits on top of the plan's normal monthly credit allowance. The cap scales with each plan: it's a share of that plan's total monthly credits, so higher tiers get a larger weekly premium allowance.

| DevPass plan | Premium fair-use cap (per week) |
| ------------ | ------------------------------- |
| **Lite**     | 12% of monthly credits          |
| **Pro**      | 15% of monthly credits          |
| **Max**      | 18% of monthly credits          |

<Callout type="info">
  Within DevPass, the weekly cap applies only to **premium** models. Standard
  models are limited only by the plan's credit balance, not by the fair-use
  window.
</Callout>

Once a DevPass plan reaches its weekly premium cap, premium requests are paused until the current 7-day window ends — usage does not gradually free up over time. The reset countdown runs from when the window opened, not from when you hit the cap, so the full allowance can return anywhere from moments to 7 days after you reach the limit. Standard models keep working normally the whole time. Upgrading the DevPass plan raises the weekly cap.

The fair-use window is a limiter on the plan allowance, not on your own money: with [pay-as-you-go overflow](https://docs.llmgateway.io/learn/payg-overflow) enabled and a positive credits balance, premium requests past the weekly cap keep flowing and bill that balance at provider rates — mid-cycle with monthly allowance remaining, or after the monthly pool is fully used alike. Standard models keep drawing from the plan allowance as usual.

## Reset Passes [#reset-passes]

You don't have to wait out the window: redeeming a [**Reset Pass**](https://docs.llmgateway.io/learn/reset-passes) instantly restores the full weekly premium allowance and starts a fresh 7-day window. A pass removes the weekly limit only — it grants no extra credits, so premium usage after a reset still draws from the plan's monthly allowance. Pro includes 1 pass per billing cycle and Max includes 2; extra passes are a one-time purchase from the DevPass dashboard. See the [Reset Passes page](https://docs.llmgateway.io/learn/reset-passes) for how redemption, included passes, and billing work.

## Provider routing on DevPass [#provider-routing-on-devpass]

DevPass plans always use the gateway's [smart routing](https://docs.llmgateway.io/features/routing): request a model by its plain id (e.g. `claude-sonnet-5`) and the gateway picks the best provider in real time based on uptime, throughput, price, and prompt-cache support. [Sticky session routing](https://docs.llmgateway.io/features/routing#sticky-session-routing) still keeps each coding session on one provider automatically, so upstream prompt caches stay warm across a conversation.

<Callout type="warn">
  **Provider pinning is not available on DevPass.** Provider-prefixed model ids
  such as `openai/gpt-4o` — which pin a request to one specific provider on
  pay-as-you-go — are rejected with a `403` on DevPass coding plans, as is
  custom provider routing. If you need to target an exact provider or region,
  use [pay-as-you-go credits](https://llmgateway.io) with the LLM Gateway API,
  where [provider-specific routing](https://docs.llmgateway.io/features/routing#provider-specific-routing)
  is fully supported.
</Callout>


# Model Usage
URL: https://docs.llmgateway.io/learn/model-usage


The Model Usage page shows how your API requests are distributed across different LLM models over time.

## Filters [#filters]

Two filters let you narrow down the data:

* **API Key** — Select a specific API key or view usage across all keys
* **Date range** — Choose a time period to analyze

## Usage Chart [#usage-chart]

The main chart displays a time-series breakdown of requests per model. Each model is represented by a different color, making it easy to see:

* Which models are used most frequently
* How usage patterns change over time
* Whether usage is concentrated on a single model or spread across many

This page is useful for understanding your model distribution and identifying opportunities to optimize costs by switching to more cost-effective models for certain workloads.


# Models
URL: https://docs.llmgateway.io/learn/models


The Models page shows every model available to your organization in one directory: the full LLM Gateway catalog plus any models defined for your own [custom providers](https://docs.llmgateway.io/features/custom-providers). When your organization enforces a [provider compliance policy](https://docs.llmgateway.io/features/compliance), the page also shows exactly which models are eligible to route to — and why the rest are blocked.

Every organization member can browse this page, including project-scoped `developer` members, so your whole team can see which providers and models are available without needing admin access. Management actions are only visible to owners and admins.

## The directory [#the-directory]

The directory is the same table you know from the public [models page](https://llmgateway.io/models), with one row per provider mapping:

| Column                 | Description                                                             |
| ---------------------- | ----------------------------------------------------------------------- |
| **Provider**           | The provider serving the model; custom providers carry a `Custom` badge |
| **Model ID**           | The model id; the copy button copies the exact string to request        |
| **Input / Output $/M** | Prices per million tokens (per-request/per-character where applicable)  |
| **Cache Read $/M**     | Cached input price where supported                                      |
| **Features**           | Capability icons: streaming, vision, tools, reasoning, JSON output, …   |

Search matches model names, ids, aliases, and providers. The **Filters** panel offers the same controls as the public directory — use case, capabilities, provider, price and context ranges — plus two controls specific to this page:

* **Source** — show **Catalog & custom** (default), only **Catalog models**, or only **Custom models**. The selector appears once your organization has custom models.
* **Eligible only** — hide models blocked by your organization's compliance policy.

### Lifecycle status [#lifecycle-status]

Every provider mapping carries exactly one lifecycle status, shown as a badge on
the mapping and selectable from the **Status** chips in the filter panel:

| Status          | Meaning                                                                  |
| --------------- | ------------------------------------------------------------------------ |
| **Deprecated**  | Still routes; the provider has announced a sunset, so migrate soon       |
| **Scheduled**   | Still routes, but deactivation is set for a date inside the next 90 days |
| **Deactivated** | No longer routes; requests return errors                                 |

Status resolves by urgency — deactivated beats scheduled beats deprecated — so a
mapping never carries two, and a deactivation further out than the 90-day notice
window stays plain active rather than warning early. The chips are single-select
and write to `?status=`, so a filtered view can be shared.

With no chip selected, the directory lists everything routable and hides
mappings already past their deprecation or deactivation date.

### Time-based pricing [#time-based-pricing]

A few provider mappings bill different rates depending on when the request
arrives. Those cards carry a **Time-based pricing** toggle: switch between
**Peak** and **Off-peak** to see each set of input, cached-input, and output
prices, with the schedule that decides which one applies spelled out underneath —
the peak hours, their time zone, and any days that are off-peak all day.

Billing follows the same schedule automatically at request time; there is
nothing to set on your side. To pay off-peak rates, move batch or backfill work
into the off-peak window.

## Compliance eligibility [#compliance-eligibility]

When your organization is on the Enterprise plan and its provider compliance policy is enabled, every row is evaluated with the same fail-closed rules the gateway enforces at request time — certifications (SOC 2, ISO 27001, GDPR), data-policy requirements (no training on prompts, no prompt logging), country restrictions, and provider/model allow and deny lists.

Rows that fail render greyed out with a red ban icon; hover it to see the exact reasons, such as "No SOC 2 Type 2 report" or "Not on the allowed-providers list". Custom providers are evaluated against their [compliance attestation](https://docs.llmgateway.io/features/custom-providers#compliance-attestation) — without an attestation on file, all requests through them are blocked while a policy is active.

<Callout type="info">
  Eligibility marking requires an active compliance policy, available on the
  [**Enterprise plan**](https://llmgateway.io/enterprise). Without one, the
  directory simply lists all models. Configure the policy on the
  [Compliance](https://docs.llmgateway.io/features/compliance) page.
</Callout>

## Custom models [#custom-models]

Models from your organization's custom-model catalog appear in the same table, addressed as `<provider>/<model>` (e.g. `internal-vllm/llama-4-maverick`) — the exact model string to send to the gateway. Their pricing, context size, and capability flags come from the catalog entries you define.

Owners and admins additionally see the **Custom model catalog** section below the directory:

* **Custom provider** — pick which custom provider key's catalog to manage
* **Only allow catalog models** — when on, requests through the provider are limited to the models defined in its catalog so cost and context limits are always enforced
* **Compliance attestation** — record the compliance posture of the infrastructure behind the provider; the policy evaluates it with the same fail-closed rules as catalogue providers
* **Add / edit / delete custom models** — define per-model pricing, context and output limits, and capabilities

<Callout type="info">
  The custom model catalog requires the [**Enterprise
  plan**](https://llmgateway.io/enterprise). Custom provider keys themselves are
  created on the [Provider Keys](https://docs.llmgateway.io/learn/provider-keys) page.
</Callout>


# Organization Analytics
URL: https://docs.llmgateway.io/learn/org-analytics


Organization Analytics rolls usage up across **every project in your organization** into one view. Where the project [Analytics](https://docs.llmgateway.io/learn/analytics) page answers "where does this project's spend go?", this page answers it for the whole org — and lets you pivot the breakdown by model, **project**, or API key.

<Callout type="info">
  Organization Analytics is an **Enterprise** feature, available to organization
  **owners and admins**. On lower plans the page shows an upgrade prompt
  instead, and Enterprise-only items are flagged in the sidebar.
</Callout>

Open it from the **Analytics** item under **Organization** in the sidebar. Like the rest of the dashboard, it respects the shared date-range picker, so every chart and stat reflects the same window.

## Summary [#summary]

Three cards at the top total the selected range across the whole organization:

| Card            | What it totals                         |
| --------------- | -------------------------------------- |
| **Total spend** | Combined cost of every project, in USD |
| **Requests**    | Total requests routed across the org   |
| **Tokens**      | Total tokens (input + output)          |

## Breakdown [#breakdown]

A single **group-by** control switches what the two charts below break the usage down by:

| Group by    | What each series represents                                                     |
| ----------- | ------------------------------------------------------------------------------- |
| **Model**   | Canonical model, collapsed across providers (Qwen via any provider is one line) |
| **Project** | One project in the organization                                                 |
| **API key** | One API key (by its description)                                                |

For each grouping you get the same two charts:

* **Over time** — a stacked area chart of the top series across the date range, with **Cost / Requests / Tokens** tabs.
* **Ranking** — a horizontal bar chart of the top series for the range, with the same metric tabs and the running totals for the window.

Switch to **Project** to see which teams or workloads drive the bill, **Model** for the org-wide model footprint, or **API key** when usage runs through services rather than people.

## How the data is computed [#how-the-data-is-computed]

Both charts read from the same pre-aggregated hourly rollups the rest of the dashboard uses — there is no separate analytics pipeline and no scan over raw request logs, so the page stays fast over any range. Aggregation happens per time bucket, so totals line up with the project [Analytics](https://docs.llmgateway.io/learn/analytics) and [Usage & Metrics](https://docs.llmgateway.io/learn/usage-metrics) pages for the same window.

For a single project's breakdown, see [Analytics](https://docs.llmgateway.io/learn/analytics). For a per-person view of org spend, see [Member Analytics](https://docs.llmgateway.io/learn/member-analytics).


# Org Preferences
URL: https://docs.llmgateway.io/learn/org-preferences


The Org Preferences page contains settings for your organization's identity and billing information.

## Organization Name [#organization-name]

Update your organization's display name. This name appears throughout the dashboard and in billing communications.

## Billing Email [#billing-email]

Set or update the email address used for billing-related communications, including receipts, invoices, and payment notifications.

## Billing Information [#billing-information]

Configure your organization's billing details for invoices:

| Field                              | Description                                                              |
| ---------------------------------- | ------------------------------------------------------------------------ |
| **Email Address**                  | Primary email for billing communications                                 |
| **Company Name** (optional)        | Your company or organization name for invoices                           |
| **Billing Address**                | Street address, city, state/province, ZIP code, and country              |
| **Tax ID / VAT Number** (optional) | Your tax identification or VAT number for tax-compliant invoices         |
| **Invoice Notes** (optional)       | Custom notes to include on invoices (e.g., PO numbers, department codes) |


# Pay-As-You-Go Overflow
URL: https://docs.llmgateway.io/learn/payg-overflow


By default a DevPass plan is a hard cap: once the monthly allowance is fully used, requests are rejected until renewal. **Pay-as-you-go overflow** is an opt-in on the DevPass **Usage** page that keeps requests flowing past the allowance — extra usage bills a credits balance at provider rates, with the same API key and the same coding agents. The Usage page also carries the full allowance meters, the [weekly premium allowance](https://docs.llmgateway.io/learn/model-categories), [Reset Passes](https://docs.llmgateway.io/learn/reset-passes), and the model usage chart.

<Callout type="info">
  Overflow is **off by default** — until you enable it, nothing ever charges
  past your subscription price. It never touches your plan allowance either:
  credits cover only usage the plan itself wouldn't serve — spend past the
  monthly pool, and premium usage past the weekly cap.
</Callout>

## When overflow applies [#when-overflow-applies]

| Situation                                   | Overflow off (default)       | Overflow on                        |
| ------------------------------------------- | ---------------------------- | ---------------------------------- |
| Within the monthly allowance                | Plan credits                 | Plan credits                       |
| Monthly allowance exhausted                 | Requests rejected with `402` | Requests bill your credits balance |
| Weekly premium cap hit, allowance remaining | Reset Pass or wait           | Premium models bill your balance   |
| Weekly premium cap hit, allowance exhausted | Requests rejected with `402` | Premium models bill your balance   |
| Credits balance empty                       | —                            | Requests rejected with `402`       |

The weekly premium fair-use cap limits what the **plan allowance** pays for, not what your own balance can: with overflow on and a positive balance, premium requests past the cap keep flowing and bill your credits at provider rates, while standard models keep drawing from the monthly allowance as usual. A [Reset Pass](https://docs.llmgateway.io/learn/reset-passes) is still the way to keep that premium spend inside your already-paid plan allowance instead of on the balance. Once the monthly pool itself is gone, overflow covers everything.

If the allowance runs out with overflow off (or with an empty balance), the gateway's `402` says what to do:

```json
{
	"error": {
		"message": "Dev Plan credit limit reached. Upgrade your plan or wait for renewal on 8/24/2026. Or enable pay-as-you-go overflow in your DevPass dashboard to keep going past your allowance.",
		"type": "invalid_request_error",
		"code": "billing_error"
	}
}
```

## Topping up [#topping-up]

Enable overflow on the Usage page, pick a preset ($10, $25, $50, $100) or a custom amount between **$10 and $5,000**, and **Top up** — the charge goes to the card already backing your DevPass subscription, with no checkout redirect. Usage itself is metered at provider rates with no markup on tokens; a 5% platform fee applies when you buy credits (plus 1.5% for international cards). Each top-up emails a PDF invoice and appears in your billing history.

Top-ups are idempotent end to end: a double-click or a retried request reuses the same charge instead of creating a second one.

## Auto-reload [#auto-reload]

Turn on **Auto-reload** and the balance refills itself: when it falls below your threshold ($5–$1,000), your reload amount ($10–$5,000) is charged to the saved card automatically.

Guardrails worth knowing:

* **Disabling overflow always disables auto-reload** — you can never buy credits you cannot spend.
* Failed auto-reload charges back off exponentially (1h doubling up to 24h), and auto-reload switches itself off after 7 days of continuous payment failures.

<Callout type="info">
  Overflow is available on **every DevPass tier**. If you find yourself covering
  a lot of usage from the balance every cycle, a tier upgrade is usually the
  better deal: plan allowances turn every subscription dollar into $3 of usage,
  while overflow bills at cost. Compare on the [DevPass pricing
  page](https://devpass.llmgateway.io/pricing).
</Callout>


# Audio Studio
URL: https://docs.llmgateway.io/learn/playground-audio


The Audio Studio turns text into speech using text-to-speech models from ElevenLabs, OpenAI, Gemini, and Qwen. Pick a model and a voice, type your text, and play or download the result.

## Model Selection [#model-selection]

Choose from the supported text-to-speech models in the dropdown. Each model has its own voice roster, output formats, and pricing — the newest models appear first.

## Generating Audio [#generating-audio]

1. Select a text-to-speech model
2. Pick a voice for the selected model
3. Type the text you want spoken — or click one of the sample prompts on the empty state
4. Click **Generate**
5. Generated clips appear in the gallery, where you can play or download them

## Voices [#voices]

Each provider ships its own voice catalog — for example OpenAI's `alloy`, `nova`, and `onyx`, Gemini's `Kore`, `Puck`, and `Zephyr`, or ElevenLabs voices like `Sarah`, `George`, and `Charlotte`. The voice picker only shows voices valid for the selected model.

## Comparison Mode [#comparison-mode]

Enable comparison mode to send the same text to multiple models at once and hear the results side by side — useful for choosing a voice and provider before wiring up the [Speech Generation API](https://docs.llmgateway.io/features/speech-generation).

## History [#history]

Generated clips are saved to your audio history so you can revisit, replay, or delete them later.


# Group Chat
URL: https://docs.llmgateway.io/learn/playground-group


The Group Chat page lets you add multiple AI models to a conversation where they discuss and build on each other's responses, creating a dynamic multi-model dialogue.

## How It Works [#how-it-works]

1. Add 2–5 different AI models to the conversation
2. Enter an initial prompt or question to kick off the discussion
3. Click **Start Conversation** to begin
4. Models take turns responding to each other in sequence
5. Each model builds on the previous responses, creating a dynamic conversation
6. You can stop the conversation at any time and start a new one

## Use Cases [#use-cases]

* **Model evaluation** — Compare how different models approach the same topic
* **Brainstorming** — Get diverse perspectives from multiple AI models
* **Debate** — Watch models discuss pros and cons of a topic
* **Research** — Gather multi-model analysis of complex questions


# Image Studio
URL: https://docs.llmgateway.io/learn/playground-image


The Image Studio lets you generate images using AI models through an intuitive interface. Select a model, describe what you want, and get results instantly.

## Model Selection [#model-selection]

Choose from supported image generation models in the dropdown. Each model has different capabilities, resolutions, and pricing.

## Generating Images [#generating-images]

1. Select an image generation model
2. Type a description of the image you want
3. Click send to generate
4. Generated images appear in the conversation

## Image Count [#image-count]

You can generate 1, 2, or 4 images at once. Multiple images are displayed in a grid layout.

## Resolution Options [#resolution-options]

Available resolutions depend on the selected model. Common options include 1K, 2K, and 4K.


# Voice Calls
URL: https://docs.llmgateway.io/learn/playground-realtime


Voice Calls turns Lounge into a phone line to a model. Pick a realtime model and a voice, start the call, and talk — the model listens, answers out loud, and the transcript builds as you go. Every call runs over the same [Realtime API](https://docs.llmgateway.io/features/realtime) your own apps can use.

## Starting a Call [#starting-a-call]

1. Pick a model from the selector — only models with an active realtime mapping appear, newest first
2. Pick a **Voice** — the voice list comes from the selected model, so it changes when you switch models
3. Click **Start call** and allow microphone access when the browser asks

The status label on the right walks through the handshake — preparing audio, requesting the microphone, authorizing, connecting, configuring — and settles on **Live** with a call timer next to it. The model and voice are locked for the duration of the call; end it to change either.

Your browser never sees an API key. The page mints a short-lived client secret server-side for the call, and the audio itself is streamed as 24 kHz PCM with echo cancellation and noise suppression on.

## During the Call [#during-the-call]

The model uses semantic voice activity detection, so there is no push-to-talk: stop speaking and it answers, start speaking again and it stops to listen. An interrupted answer stays in the transcript marked `(interrupted)`.

| Control                  | What it does                                                |
| ------------------------ | ----------------------------------------------------------- |
| **Mute** / **Unmute**    | Stops sending your microphone audio without ending the call |
| **End call**             | Hangs up and saves the call to your history                 |
| Voice activity indicator | Shows who currently has the floor, and your mute state      |

Below the controls, a usage line totals the call: how many responses it took, input and output tokens, and how many of those were audio tokens.

<Callout type="info">
  Voice calls are billed to a pay-as-you-go organization with credits, so the
  organization switcher on this page only lists those — DevPass and Lounge
  membership workspaces can't place calls. Without one selected, the page asks
  you to create one in the dashboard first.
</Callout>

## Call History [#call-history]

Ended calls are saved per organization and listed in the sidebar, grouped by date, with each call's duration and time. Click one to reopen its transcript, along with the model and voice it used and its token totals. Use the &#x2A;*⋮** menu on a call to rename or delete it, and **New Call** to get back to a fresh screen.

Calls are titled from your opening line, so history reads like a list of topics. A call you leave mid-conversation — by navigating away or closing the tab — is still saved with whatever was said up to that point.

## Using It From Your Own App [#using-it-from-your-own-app]

The same endpoint backs the [Realtime API](https://docs.llmgateway.io/features/realtime): connect a WebSocket to `wss://api.llmgateway.io/v1/realtime`, authenticate with an [API key](https://docs.llmgateway.io/learn/api-keys), and use the standard OpenAI realtime event protocol. For browser clients, mint an ephemeral client secret from your backend instead of shipping a key:

```bash
curl -X POST "https://api.llmgateway.io/v1/realtime/client_secrets" \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": { "anchor": "created_at", "seconds": 120 },
    "session": { "type": "realtime", "model": "gpt-realtime" }
  }'
```

Calls made here appear in your [activity feed](https://docs.llmgateway.io/learn/activity) and [analytics](https://docs.llmgateway.io/learn/analytics) like any other request, with audio input and output costs broken out.


# Video Studio
URL: https://docs.llmgateway.io/learn/playground-video


The Video Studio lets you generate videos using AI models. Select a model, describe what you want, and get video results.

## Model Selection [#model-selection]

Choose from supported video generation models in the dropdown. Each model has different capabilities, resolutions, and pricing.

## Generating Videos [#generating-videos]

1. Select a video generation model
2. Type a description of the video you want
3. Click send to generate
4. Generated videos appear in the conversation

## Resolution Options [#resolution-options]

Available resolutions depend on the selected model.


# Lounge Chat
URL: https://docs.llmgateway.io/learn/playground


Lounge is a standalone chat app for talking to any LLM through a conversational interface. You can select any supported model, adjust parameters, and see responses in real time.

## Model Selection [#model-selection]

Use the dropdown at the top to pick a model and provider. The **Auto Route** option automatically selects the best provider based on availability and cost.

## Chat Interface [#chat-interface]

* Type your message in the input field at the bottom
* Click the send button or press Enter to submit
* Responses stream in real time
* Previous conversations appear in the sidebar

## Prompt Suggestions [#prompt-suggestions]

When starting a new chat, category tabs help you pick a prompt:

* **Create** — Content generation prompts
* **Explore** — Research and analysis prompts
* **Code** — Programming and development prompts
* **Image gen** — Image generation prompts

## Sidebar [#sidebar]

The left sidebar shows your chat history. Click **+ New Chat** to start a fresh conversation, or select a previous chat to continue it.

## Comparison Mode [#comparison-mode]

Toggle **Comparison mode** in the top-right to send the same prompt to multiple models side by side. See the [Group Chat](https://docs.llmgateway.io/learn/playground-group) page for details.

## Image Studio [#image-studio]

Click **Image Studio** in the sidebar to switch to the image generation interface. See the [Image Studio](https://docs.llmgateway.io/learn/playground-image) page for details.


# Policies
URL: https://docs.llmgateway.io/learn/policies


The Policies page lets you configure organization-wide policies that govern how your data is handled.

## Data Retention [#data-retention]

Control how long your request logs and activity data are stored. The retention period depends on your plan:

| Plan           | Retention Period |
| -------------- | ---------------- |
| **Free**       | 30 days          |
| **Enterprise** | Custom           |

After the retention period expires, request logs and associated data are automatically deleted.

The retention level itself is configurable on standard pay-as-you-go organizations only. DevPass and chat subscriptions are always metadata only — their request and response payloads are not retained, aside from Responses API stored responses, which are kept for 30 days independently of the retention level.

Learn more about data retention in the [Data Retention feature docs](https://docs.llmgateway.io/features/data-retention).


# Preferences
URL: https://docs.llmgateway.io/learn/preferences


The Preferences page contains project-level settings that control how your project behaves.

## Project Name [#project-name]

Update the display name for your project. This name appears in the sidebar and throughout the dashboard.

## Project Mode [#project-mode]

Configure how your organization handles projects. This setting determines the routing and isolation behavior for API requests within the project.

## Caching [#caching]

Enable or configure response caching for API requests. When enabled, identical requests will return cached responses instead of making new calls to the provider, saving both time and cost.

Response caching must be disabled for every project before the organization can enable [zero data retention](https://docs.llmgateway.io/features/compliance). While ZDR is active, response caching cannot be enabled and provider prompt-cache markers are stripped.

Learn more about caching in the [Caching feature docs](https://docs.llmgateway.io/features/caching).

## Danger Zone [#danger-zone]

The Danger Zone section contains irreversible actions:

* **Archive Project** — Permanently archive the project. This action cannot be undone. Archived projects stop processing requests and their API keys become inactive.


# Provider Keys
URL: https://docs.llmgateway.io/learn/provider-keys


The Provider Keys page lets you add your own API keys from LLM providers (OpenAI, Anthropic, Google, etc.) to route requests directly through your accounts without additional gateway fees. The only optional charge is [data storage](https://docs.llmgateway.io/features/data-retention#storage-pricing): if your organization enables full data retention, stored requests are billed at $0.01 per 1M tokens from your credits.

## Adding a Provider Key [#adding-a-provider-key]

Click **Add Provider Key** to configure a new key:

* **Provider** — Select which provider this key belongs to
* **Custom name** — An optional label to identify the key
* **API key** — Your provider's API key
* **Base URL** — Optional custom endpoint (useful for Azure OpenAI or custom deployments)
* **Allowed models** — Optional allowlist restricting which models the key serves
* **Max spend** — Optional USD cap; the key is disabled once the spend attributed to it reaches the limit

## Provider Keys List [#provider-keys-list]

Each configured key shows:

| Field           | Description                                        |
| --------------- | -------------------------------------------------- |
| **Provider**    | The LLM provider (e.g., OpenAI, Anthropic)         |
| **Custom name** | Your label for the key                             |
| **Status**      | Active, inactive, or deleted                       |
| **Base URL**    | Custom endpoint if configured                      |
| **Token**       | Masked key with only the last 4 characters visible |

<Callout type="info">
  Provider credentials are encrypted at rest because LLM Gateway must decrypt
  them to call the upstream provider. Gateway API keys work differently: new and
  rolled secrets are retained only as keyed HMAC-SHA-256 fingerprints, and
  authentication compares fingerprints instead of recovering the secret.
</Callout>

## Actions [#actions]

For each provider key:

* **Edit** — Update the key name, value, or base URL
* **Restrict models** — Choose which models the key may serve
* **Set spend limit** — Cap the spend attributed to the key
* **Deactivate** — Temporarily disable the key without deleting it
* **Delete** — Permanently remove the key

## Restricting a Key to Specific Models [#restricting-a-key-to-specific-models]

Provider accounts often only have some of a provider's models enabled. Set
**Allowed models** on a key to list exactly the models it may serve — routing
then skips that key for anything else instead of sending a request the upstream
account will reject. Leave the list empty to serve every model the provider
offers.

The restriction also decides how a key is checked when you save it: a restricted
key is validated against one of its own allowed models rather than the
provider's default validation model, so a key with partial model access still
validates.

<Callout type="info">
  In hybrid mode, a model excluded by the restriction falls back to your credits
  rather than failing, so restricting a key never takes a model away from your
  projects.
</Callout>

<Callout type="info">
  When you use your own provider keys, requests are routed directly to the
  provider. You pay the provider's standard rates with no additional gateway
  markup — the only LLM Gateway charge is the optional [data
  storage](https://docs.llmgateway.io/features/data-retention#storage-pricing) cost when full data
  retention is enabled.
</Callout>


# Referrals
URL: https://docs.llmgateway.io/learn/referrals


The Referrals page lets you earn credits by inviting others to use LLM Gateway.

## Eligibility [#eligibility]

To unlock the referral program, your organization must have at least **$100 in total credit top-ups**. Before reaching this threshold, the page shows:

* A progress bar showing your progress toward $100
* The remaining amount needed to unlock
* An explanation of the 1% earnings model

## Referral Dashboard [#referral-dashboard]

Once eligible, the page shows:

### Your Referral Link [#your-referral-link]

A unique shareable link tied to your organization. Click the copy button to copy it to your clipboard and share it with others.

### Your Stats [#your-stats]

| Stat               | Description                                           |
| ------------------ | ----------------------------------------------------- |
| **Users Referred** | Total number of users who signed up through your link |
| **Total Earnings** | Total credit amount earned from referrals             |

### How It Works [#how-it-works]

1. **Share Your Link** — Send your referral link to others
2. **They Sign Up** — They create an LLM Gateway account using your link
3. **Earn Credits** — You earn 1% of their spending as credits

Credits are automatically added to your organization balance.


# Reset Passes
URL: https://docs.llmgateway.io/learn/reset-passes


DevPass plans include a [weekly fair-use allowance](https://docs.llmgateway.io/learn/model-categories) for premium frontier models. A **Reset Pass** restores that full allowance the moment you redeem it — no waiting for the rolling 7-day window — and a fresh window starts with your next premium request. The pass card lives on the DevPass dashboard directly under the premium allowance meter, styled as a stamped visa extension: each pass you hold appears as a stamp.

<Callout type="warn">
  A Reset Pass removes the weekly limit — it does **not** add credits. Premium
  usage after a reset draws from your plan's monthly credit allowance exactly as
  before; the pass just lets you keep using premium models now instead of
  waiting for the window. With [pay-as-you-go overflow](https://docs.llmgateway.io/learn/payg-overflow)
  enabled, premium usage past the cap bills your credits balance at provider
  rates instead — a pass is what brings that spend back inside the plan
  allowance you already paid for. And once the monthly allowance itself is
  exhausted, overflow is what keeps requests flowing, not a pass.
</Callout>

<Callout type="info">
  Reset Passes exist only on **DevPass** (Lite, Pro, Max). Pay-as-you-go API
  usage has no weekly premium cap, so there is nothing to reset.
</Callout>

## Included and purchased passes [#included-and-purchased-passes]

| Plan     | Included per billing cycle | Extra passes |
| -------- | -------------------------- | ------------ |
| **Lite** | —                          | $9 each      |
| **Pro**  | 1                          | $29 each     |
| **Max**  | 2                          | $79 each     |

Two kinds of stamps appear on the card:

* **Included passes** come with Pro and Max. They refresh every billing cycle, don't roll over, and are always consumed before purchased passes.
* **Purchased passes** are one-time purchases at the tier's price, bound to the tier they were bought for — a pass purchased on Pro is redeemable while you're on Pro. They never expire, so an unused pass applies again whenever you're back on that tier, even after a cancelled plan.

## Redeeming a pass [#redeeming-a-pass]

Click **Use a pass** on the card. Your premium usage for the week is zeroed immediately and the 7-day window restarts with your next premium request. Redemption is deliberately explicit — a pass is never consumed automatically, and it is rejected while your allowance is untouched. Once you've used more than 90% of your monthly credit allowance, the dashboard also asks you to confirm: a reset restores the weekly cap, but only the remaining monthly allowance is available to spend. Once the monthly allowance is fully used, Reset Passes cannot be redeemed.

Standard models are unaffected either way: they never count against the weekly cap.

## Renewal and the weekly window [#renewal-and-the-weekly-window]

Nothing about the weekly allowance carries over between billing cycles — every monthly renewal starts you completely fresh, all at once:

* **The weekly premium allowance resets in full.** Renewal zeroes your premium usage and starts a new 7-day window at the renewal moment, alongside the monthly credit reset. You never begin a cycle mid-window or with a partially used allowance.
* **Included passes replenish.** Pro's 1 and Max's 2 included passes are restored every cycle; unused ones don't roll over. Purchased passes are unaffected — they persist until redeemed.

An immediate tier upgrade restarts your billing cycle and performs the same full reset.

Between renewals the window is **rolling, not on a fixed schedule**. When a 7-day window ends, the next one starts with your next premium request — and a redeemed pass clears the window entirely, so your next premium request starts a fresh 7 days. There is no weekly reset schedule that redeeming late (or early) can shift into the next cycle.

In practice, a 30-day cycle of steady premium usage fits about five weekly windows (starting around days 0, 7, 14, 21, and 28) before any passes. On Max, whose weekly cap is 18% of the monthly allowance, those five windows plus a single included pass already unlock more than the entire monthly credit pool — so even spending exclusively on premium models needs no front-loaded pass redemptions or timing strategy. Redeem when you actually hit the cap.

If you hit the cap from a coding agent, the gateway's `402` response points here:

```json
{
	"error": {
		"message": "You've used your weekly allowance for premium-tier models on the pro plan. Redeem a Reset Pass from your dashboard for an instant reset, upgrade for a higher allowance, or use any standard model now. Resets in 6 days.",
		"type": "invalid_request_error",
		"code": "billing_error"
	}
}
```

## Buying and billing [#buying-and-billing]

**Buy a pass** shows a confirmation dialog, then charges the payment method saved on your DevPass — no checkout redirect, no subscription change, no proration. Each purchase emails a PDF invoice and appears in the DevPass billing history alongside plan charges. Purchases are paused once you've used more than 95% of your monthly credit allowance — a pass would give you almost nothing to use at that point — and open up again when your credits renew.

An unused pass can be returned: for 7 days after purchase, a purchased pass that hasn't been redeemed can be self-refunded from the billing history. The refund goes back to your payment method, the pass is removed from your passport, and your DevPass plan itself is unaffected.

<Callout type="info">
  Reaching for a Reset Pass every week? Upgrading a tier is usually the better
  deal — the next tier raises the weekly cap itself and brings a larger monthly
  allowance. Compare on the [DevPass pricing
  page](https://devpass.llmgateway.io/pricing).
</Callout>


# Payments SDK
URL: https://docs.llmgateway.io/learn/sdk-settings


The **Payments SDK** settings page lets you embed end-user payments and sessions into your own application — your end users get their own wallets, and you control markup and access. It is a payments feature, not a normal AI client SDK like the OpenAI SDK. You'll find it under **Settings → Payments SDK** for a project.

<Callout type="warn">
  **Preview — opt-in only.** Embeddable Payments is in preview and enabled on an
  opt-in basis per project. Until it is turned on for your project, this page is
  read-only and shows a preview of the settings. [Contact
  us](mailto:contact@llmgateway.io) to request access.
</Callout>

## End-user sessions [#end-user-sessions]

Turn on **Enable end-user sessions** to allow this project to mint short-lived browser session tokens for your users.

| Field               | Description                                                                                        |
| ------------------- | -------------------------------------------------------------------------------------------------- |
| **Markup percent**  | The percentage you add on top of provider cost for each end-user request (0–100%)                  |
| **Allowed origins** | The browser origins permitted to use session tokens, one per line (e.g. `https://app.example.com`) |

Click **Save Settings** to apply changes.

## Platform secret keys [#platform-secret-keys]

Platform secret keys are **server-side** keys used to mint end-user sessions. Keep them on your backend — never expose them in the browser.

* **Create Live Key** — A production key. Top-ups made with it use live billing.
* **Create Test Key** — A sandbox key. Top-ups use the Stripe sandbox, so you can build and test without real charges.

<Callout type="warn">
  A secret key is shown **only once** at creation time. Copy it immediately — it
  won't be displayed again. Platform secrets and ephemeral session tokens are
  stored only as keyed fingerprints. If you lose a key, revoke it and create a
  new one.
</Callout>

Each key in the list shows its description, a **test** badge when applicable, its status, and a masked token. Use **Revoke** to permanently disable a key.

<Callout type="info">
  For the full SDK integration guide — server, client, and React components —
  see the [Embeddable Payments feature docs](https://docs.llmgateway.io/features/embeddable-payments).
</Callout>


# Security Events
URL: https://docs.llmgateway.io/learn/security-events


The Security Events page shows all guardrail violations detected across your organization, helping you monitor content safety and policy enforcement. Violations from projects that [override the organization guardrails](https://docs.llmgateway.io/learn/guardrails) appear here too, so this page stays the single place to review enforcement.

<Callout type="info">
  Security Events are available on the [**Enterprise
  plan**](https://llmgateway.io/enterprise). Owner or Admin role is required.
</Callout>

## Stats Cards [#stats-cards]

Four summary cards at the top:

| Card                 | Description                                   |
| -------------------- | --------------------------------------------- |
| **Total Violations** | All-time violation count                      |
| **Last 24 Hours**    | Violations in the past day                    |
| **Blocked**          | Number of requests that were blocked          |
| **Redacted**         | Number of requests where content was redacted |

## Filters [#filters]

Narrow down the events list:

* **Action** — Filter by Blocked, Redacted, Warned, or All actions
* **Category** — Filter by Prompt Injection, Jailbreak, PII Detection, Secrets, Blocked Terms, Custom Regex, or Topic Restriction

## Violations List [#violations-list]

Each violation entry shows:

| Field               | Description                                          |
| ------------------- | ---------------------------------------------------- |
| **Timestamp**       | When the violation occurred                          |
| **Rule name**       | Which guardrail rule was triggered                   |
| **Category**        | The type of violation (shown as a badge)             |
| **Action**          | What action was taken (Blocked, Redacted, or Warned) |
| **Matched pattern** | The content that triggered the rule                  |

The list supports pagination with a **Load More** button for viewing older events.


# Structured outputs
URL: https://docs.llmgateway.io/learn/structured-outputs


LLM Gateway exposes **two distinct JSON capabilities**, and they are kept
separate on every surface: the models API, the [models
directory](https://llmgateway.io/models), the model detail pages, and the
gateway's request validation.

## Soft JSON output (`json_output`) [#soft-json-output-json_output]

A model with soft JSON output can be **nudged** into emitting JSON — typically
via `response_format: { "type": "json_object" }` or prompt guidance. There is
no server-side schema guarantee: the model may still produce off-schema or
malformed JSON, so the consumer's parser must be able to handle it.

## Strict JSON output schema (`structured_outputs`) [#strict-json-output-schema-structured_outputs]

A model with strict JSON output schema is served by an **upstream provider
that natively enforces schema-guided decoding** (for example OpenAI structured
outputs or vLLM guided decoding). You provide a JSON schema via
`response_format: { "type": "json_schema", "json_schema": { ... } }` and the
provider guarantees the output conforms to it.

<Callout type="info">
  The gateway only **declares** this capability — it never emulates schema
  enforcement (no prompt-and-validate adapter). A model whose provider does not
  natively support `json_schema` rejects such requests with `400 does not
  	support JSON schema output mode`. That rejection is **by design**, not a
  missing feature.
</Callout>

## Field mapping [#field-mapping]

| Models API field     | Catalogue flag     | Meaning                                       |
| -------------------- | ------------------ | --------------------------------------------- |
| `json_output`        | `jsonOutput`       | Soft: nudged JSON, no schema guarantee        |
| `structured_outputs` | `jsonOutputSchema` | Strict: upstream provider enforces the schema |

The snake\_case names are what the OpenAI-compatible [`/v1/models`](https://docs.llmgateway.io/learn/models) endpoint exposes;
the camelCase names are used in the model catalogue and the directory UI.

The two tiers are **independent per provider mapping**: a provider may support
strict `json_schema` without soft `json_object` (for example Runware or
Perplexity declare `structured_outputs` without `json_output`), or soft
`json_object` without strict schema. The gateway routes each mode by its own
flag — `json_schema` requests only require `structured_outputs`, they are never
emulated or silently downgraded.

## How to verify a model [#how-to-verify-a-model]

Probe a model with a strict `json_schema` request:

```bash
curl -s https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLMGATEWAY_API_KEY" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Reply JSON: {\"ok\":true}" }],
    "max_tokens": 16,
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "probe",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": { "ok": { "type": "boolean" } },
          "required": ["ok"],
          "additionalProperties": false
        }
      }
    }
  }'
```

* `200` → the upstream provider enforces the schema; the model should declare `structured_outputs: true`.
* `400 does not support JSON schema output mode` → soft-only; the model should expose `json_output` without `structured_outputs`.

The probe above can be run for any model listed in `GET /v1/models`; a
declared flag that does not match the observed status is a catalog bug and
should be reported against the repository.


# Team
URL: https://docs.llmgateway.io/learn/team


The Team page has two views: **Members** for organization membership and personal controls, and **Teams** for policy shared by groups of developers.

## Members [#members]

The Members view shows each person's role, assigned organization team, projects, and limits. Enterprise owners and admins can also compare cost, tokens, requests, and API-key counts for the selected period.

Each row has actions to:

* Open the member's details and usage
* Change their role and project access
* Set a personal API-key or spending budget
* Add personal IAM rules
* Remove them from the organization

Personal IAM rules apply to every key created by that member. A key can narrow those rules, but cannot expand access beyond them. See [Member-level IAM rules](https://docs.llmgateway.io/features/api-keys#member-level-iam-rules) for rule behavior and [Member IAM rules](https://docs.llmgateway.io/features/master-keys#member-iam-rules) for the management API.

### Roles [#roles]

| Role          | Permissions                                                                                               |
| ------------- | --------------------------------------------------------------------------------------------------------- |
| **Owner**     | Full access, including team management, billing, and organization settings                                |
| **Admin**     | Can manage members, projects, and API keys, but cannot access billing or modify owners                    |
| **Developer** | Can view and use assigned projects and API keys, but cannot manage the organization or other team members |

### Invite a member [#invite-a-member]

Click **Add Member**, enter an email address, and choose a role. A Developer requires Enterprise access and at least one project grant. The invitation remains under **Pending Invitations** until it is accepted or revoked.

Pending invitations reserve a seat. The Members card shows the current seat count and plan limit, and **Add Member** is disabled when the limit is reached.

## Organization teams [#organization-teams]

Open **Teams** to group developers under a shared project, IAM, and budget policy. One developer can belong to one organization team; owners and admins cannot be assigned.

<Callout type="info">
  Creating teams and changing team policy requires the [**Enterprise
  plan**](https://llmgateway.io/enterprise). If Enterprise access ends, existing
  policy remains enforced and can still be reviewed. Developers can be
  unassigned, and empty teams can be deleted.
</Callout>

Click **Create team**, give it a unique name, then click **Open** to configure it. The list shows the number of developers, project ceiling, and IAM rule count for every team.

## Configure team policy [#configure-team-policy]

| Control                           | Behavior                                                                                                                                     |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **Team identity**                 | Names the team and can mark it as the organization's **default team**. Names are unique within the organization.                             |
| **Project ceiling**               | Effective access is the intersection of the team list and each developer's personal grants. With no projects selected, user keys return 403. |
| **Per-developer budget ceilings** | Caps each developer's active keys, lifetime spend, and recurring spend. Personal and API-key limits may be stricter.                         |
| **Developers**                    | Assigns, moves, or unassigns Developers. A move replaces the previous team's policy immediately.                                             |
| **IAM policy**                    | Adds shared model, provider, pricing, or IP rules. Member and API-key rules run afterward and can only narrow access.                        |

Unassigning a developer restores their personal project, IAM, and budget settings. Unassign every developer before deleting a team.

### Default team [#default-team]

One team per organization can be marked as the default. Developers who join — by invite, direct add, SSO domain auto-join, or SCIM provisioning — are assigned to it automatically unless a SCIM group mapping places them elsewhere; explicit manual assignments always win. When enabling the default you can optionally assign every developer currently without a team in one step. Moving or removing the flag only changes future joins: current members keep their team.

## Sync groups from Microsoft Entra ID [#sync-groups-from-microsoft-entra-id]

Microsoft Entra groups reach LLM Gateway through SCIM group provisioning:

1. Create or choose a security group in Entra and assign its users.
2. In the LLM Gateway enterprise application, configure SCIM provisioning and include that group in the provisioning scope.
3. In LLM Gateway, open **SSO**, then map the exact Entra group name to an existing organization team.
4. When Entra sends the group and membership update, matching Developers are assigned to that team. Removing a user from the Entra group removes the synced assignment.

The team detail page identifies assignments managed by directory sync. Manually unassigning a synced developer lasts only until the next SCIM update; an explicit manual assignment to another team is preserved. If several mapped groups apply, the first group name alphabetically wins. Owners and admins never inherit team policy.

Follow [Configure Microsoft Entra ID](https://docs.llmgateway.io/features/sso/entra) for the complete Entra application and SCIM setup. For other identity providers, see [Single Sign-On](https://docs.llmgateway.io/features/sso).

Team, member, IAM, budget, and directory-sync changes are recorded in [Audit Logs](https://docs.llmgateway.io/learn/audit-logs).


# Transactions
URL: https://docs.llmgateway.io/learn/transactions


The Transactions page shows a complete history of all financial transactions in your organization.

## Transaction History [#transaction-history]

Each transaction entry includes:

| Field           | Description                              |
| --------------- | ---------------------------------------- |
| **Date**        | When the transaction occurred            |
| **Type**        | The transaction type (see below)         |
| **Credits**     | Number of credits added or deducted      |
| **Total Paid**  | The dollar amount charged                |
| **Status**      | Current state of the transaction         |
| **Description** | Additional details about the transaction |

## Transaction Types [#transaction-types]

| Type                    | Description                         |
| ----------------------- | ----------------------------------- |
| **Credit Top-up**       | Manual or automatic credit purchase |
| **Credit Refund**       | Credits refunded to your account    |
| **Subscription Start**  | New plan subscription started       |
| **Subscription Cancel** | Plan subscription canceled          |
| **Subscription End**    | Plan subscription period ended      |

## Status Badges [#status-badges]

* **Completed** — Transaction processed successfully
* **Pending** — Transaction is being processed
* **Failed** — Transaction could not be completed


# Usage & Metrics
URL: https://docs.llmgateway.io/learn/usage-metrics


The Usage & Metrics page provides comprehensive analytics through five tabs, giving you deep insight into your LLM API usage patterns.

## Filters [#filters]

* **API Key** — Filter metrics by a specific API key or view all
* **Date range** — Select the time period (defaults to last 7 days)

## Tabs [#tabs]

### Requests [#requests]

A time-series chart showing request volume over the selected period. Use this to identify traffic patterns, peak usage times, and growth trends.

### Models [#models]

A table showing your top-used models ranked by request count. For each model you can see:

* Total requests
* Token consumption
* Associated costs

This helps you understand which models drive the most usage and cost.

### Errors [#errors]

A chart showing error rates over time. Track:

* Error frequency and trends
* Spikes that may indicate provider issues
* Overall reliability of your API calls

### Cache [#cache]

A chart showing your cache hit rate over time. Monitor:

* How effectively caching is reducing redundant requests
* Cache hit vs. miss ratios
* The cost savings from cached responses

### Costs [#costs]

A cost breakdown chart showing spending patterns. Analyze:

* Cost trends over time
* Cost distribution by provider or model
* Opportunities to reduce spending


# Migrate from LiteLLM
URL: https://docs.llmgateway.io/migrations/litellm


<section>
  Running your own LiteLLM proxy works—until it doesn't. Scaling, monitoring, and keeping it running becomes another job. LLM Gateway gives you the same unified API with built-in analytics, caching, and a dashboard—without the infrastructure overhead.

  ## Quick Migration [#quick-migration]

  Both services use OpenAI-compatible endpoints, so migration is a two-line change:

  ```diff
  - const baseURL = "http://localhost:4000/v1";  // LiteLLM proxy
  + const baseURL = "https://api.llmgateway.io/v1";

  - const apiKey = process.env.LITELLM_API_KEY;
  + const apiKey = process.env.LLM_GATEWAY_API_KEY;
  ```
</section>

<section>
  ## Why Teams Switch to LLM Gateway [#why-teams-switch-to-llm-gateway]

  | What You Get             | LiteLLM (Self-Hosted) | LLM Gateway            |
  | ------------------------ | --------------------- | ---------------------- |
  | OpenAI-compatible API    | Yes                   | Yes                    |
  | Infrastructure to manage | Yes (you run it)      | No (we run it)         |
  | Managed cloud option     | No                    | Yes                    |
  | Analytics dashboard      | Basic                 | Per-request detail     |
  | Response caching         | Manual setup          | Built-in, automatic    |
  | Cost tracking            | Via callbacks         | Native, real-time      |
  | Provider key management  | Config file           | Web UI with rotation   |
  | Uptime & scaling         | You handle it         | 99.9% SLA (Enterprise) |

  Still want to self-host? LLM Gateway is [open source under AGPLv3](https://llmgateway.io/blog/how-to-self-host-llm-gateway)—same features, your infrastructure.

  For a detailed breakdown, see [LLM Gateway vs LiteLLM](https://llmgateway.io/compare/litellm).
</section>

<section>
  ## Migration Steps [#migration-steps]

  <Steps>
    <Step>
      ### Get Your LLM Gateway API Key [#get-your-llm-gateway-api-key]

      Sign up at [llmgateway.io/signup](https://llmgateway.io/signup) and create an API key from your dashboard.
    </Step>

    <Step>
      ### Map Your Models [#map-your-models]

      LLM Gateway supports two model ID formats:

      **Canonical Model IDs** (without provider prefix) - Uses smart routing to automatically select the best provider based on uptime, throughput, price, and latency:

      ```
      gpt-5.2
      claude-opus-4-5-20251101
      gemini-3-flash-preview
      ```

      **Provider-Prefixed Model IDs** - Routes to a specific provider with automatic failover if uptime drops below 90%:

      ```
      openai/gpt-5.2
      anthropic/claude-opus-4-5-20251101
      google-ai-studio/gemini-3-flash-preview
      ```

      This means many LiteLLM model names work directly with LLM Gateway:

      | LiteLLM Model                    | LLM Gateway Model                                                 |
      | -------------------------------- | ----------------------------------------------------------------- |
      | gpt-5.2                          | gpt-5.2 or openai/gpt-5.2                                         |
      | claude-opus-4-5-20251101         | claude-opus-4-5-20251101 or anthropic/claude-opus-4-5-20251101    |
      | gemini/gemini-3-flash-preview    | gemini-3-flash-preview or google-ai-studio/gemini-3-flash-preview |
      | bedrock/claude-opus-4-5-20251101 | claude-opus-4-5-20251101 or aws-bedrock/claude-opus-4-5-20251101  |

      For more details on routing behavior, see the [routing documentation](https://docs.llmgateway.io/features/routing).
    </Step>

    <Step>
      ### Update Your Code [#update-your-code]

      #### Python with OpenAI SDK [#python-with-openai-sdk]

      ```python
      from openai import OpenAI

      # Before (LiteLLM proxy)
      client = OpenAI(
          base_url="http://localhost:4000/v1",
          api_key=os.environ["LITELLM_API_KEY"]
      )

      response = client.chat.completions.create(
          model="gpt-4",
          messages=[{"role": "user", "content": "Hello!"}]
      )

      # After (LLM Gateway) - model name can stay the same!
      client = OpenAI(
          base_url="https://api.llmgateway.io/v1",
          api_key=os.environ["LLM_GATEWAY_API_KEY"]
      )

      response = client.chat.completions.create(
          model="gpt-4",  # or "openai/gpt-4" to target a specific provider
          messages=[{"role": "user", "content": "Hello!"}]
      )
      ```

      #### Python with LiteLLM Library [#python-with-litellm-library]

      If you're using the LiteLLM library directly, you can point it to LLM Gateway:

      ```python
      import litellm

      # Before (direct LiteLLM)
      response = litellm.completion(
          model="gpt-4",
          messages=[{"role": "user", "content": "Hello!"}]
      )

      # After (via LLM Gateway) - same model name works
      response = litellm.completion(
          model="gpt-4",  # or "openai/gpt-4" to target a specific provider
          messages=[{"role": "user", "content": "Hello!"}],
          api_base="https://api.llmgateway.io/v1",
          api_key=os.environ["LLM_GATEWAY_API_KEY"]
      )
      ```

      #### TypeScript/JavaScript [#typescriptjavascript]

      ```typescript
      import OpenAI from "openai";

      // Before (LiteLLM proxy)
      const client = new OpenAI({
      	baseURL: "http://localhost:4000/v1",
      	apiKey: process.env.LITELLM_API_KEY,
      });

      // After (LLM Gateway) - same model name works
      const client = new OpenAI({
      	baseURL: "https://api.llmgateway.io/v1",
      	apiKey: process.env.LLM_GATEWAY_API_KEY,
      });

      const completion = await client.chat.completions.create({
      	model: "gpt-4", // or "openai/gpt-4" to target a specific provider
      	messages: [{ role: "user", content: "Hello!" }],
      });
      ```

      #### cURL [#curl]

      ```bash
      # Before (LiteLLM proxy)
      curl http://localhost:4000/v1/chat/completions \
        -H "Authorization: Bearer $LITELLM_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "gpt-4",
          "messages": [{"role": "user", "content": "Hello!"}]
        }'

      # After (LLM Gateway) - same model name works
      curl https://api.llmgateway.io/v1/chat/completions \
        -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "gpt-4",
          "messages": [{"role": "user", "content": "Hello!"}]
        }'
      # Use "openai/gpt-4" to target a specific provider
      ```
    </Step>

    <Step>
      ### Migrate Configuration [#migrate-configuration]

      #### LiteLLM Config (Before) [#litellm-config-before]

      ```yaml
      # litellm_config.yaml
      model_list:
        - model_name: gpt-4
          litellm_params:
            model: gpt-4
            api_key: sk-...
        - model_name: claude-3
          litellm_params:
            model: claude-3-sonnet-20240229
            api_key: sk-ant-...
      ```

      #### LLM Gateway (After) [#llm-gateway-after]

      With LLM Gateway, you don't need a config file. Provider keys are managed in the web dashboard, or you can use the default LLM Gateway keys.

      If you want to use your own provider keys, configure them in the dashboard under Settings > Provider Keys.
    </Step>
  </Steps>
</section>

<section>
  ## Streaming Support [#streaming-support]

  LLM Gateway supports streaming identically to LiteLLM:

  ```python
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.llmgateway.io/v1",
      api_key=os.environ["LLM_GATEWAY_API_KEY"]
  )

  stream = client.chat.completions.create(
      model="openai/gpt-4",
      messages=[{"role": "user", "content": "Write a story"}],
      stream=True
  )

  for chunk in stream:
      if chunk.choices[0].delta.content:
          print(chunk.choices[0].delta.content, end="")
  ```
</section>

<section>
  ## Function/Tool Calling [#functiontool-calling]

  LLM Gateway supports function calling:

  ```python
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.llmgateway.io/v1",
      api_key=os.environ["LLM_GATEWAY_API_KEY"]
  )

  tools = [{
      "type": "function",
      "function": {
          "name": "get_weather",
          "description": "Get the weather for a location",
          "parameters": {
              "type": "object",
              "properties": {
                  "location": {"type": "string"}
              },
              "required": ["location"]
          }
      }
  }]

  response = client.chat.completions.create(
      model="openai/gpt-4",
      messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
      tools=tools
  )
  ```
</section>

<section>
  ## Removing LiteLLM Infrastructure [#removing-litellm-infrastructure]

  After verifying LLM Gateway works for your use case, you can decommission your LiteLLM proxy:

  1. Update all clients to use LLM Gateway endpoints
  2. Monitor the LLM Gateway dashboard for successful requests
  3. Shut down your LiteLLM proxy server
  4. Remove LiteLLM configuration files

  ## What Changes After Migration [#what-changes-after-migration]

  * **No servers to babysit** — We handle scaling, uptime, and updates
  * **Real-time cost visibility** — See what every request costs, broken down by model
  * **Automatic caching** — Repeated requests hit cache, reducing your spend
  * **Web-based management** — No more editing YAML files for config changes
  * **New models immediately** — Access new releases within 48 hours, no deployment needed
</section>

<section>
  ## Self-Hosting LLM Gateway [#self-hosting-llm-gateway]

  If you prefer self-hosting like LiteLLM, LLM Gateway is available under AGPLv3:

  ```bash
  git clone https://github.com/llmgateway/llmgateway
  cd llmgateway
  pnpm install
  pnpm setup
  pnpm dev
  ```

  This gives you the same benefits as LiteLLM's self-hosted proxy with LLM Gateway's analytics and caching features.

  ## Full Comparison [#full-comparison]

  Want to see a detailed breakdown of all features? Check out our [LLM Gateway vs LiteLLM comparison page](https://llmgateway.io/compare/litellm).
</section>

<footer>
  ## Need Help? [#need-help]

  * Browse available models at [llmgateway.io/models](https://llmgateway.io/models)
  * Read the [API documentation](https://docs.llmgateway.io)
  * Contact support at [contact@llmgateway.io](mailto:contact@llmgateway.io)
</footer>


# Migrate from OpenRouter
URL: https://docs.llmgateway.io/migrations/openrouter


<section>
  LLM Gateway works just like OpenRouter—same API format, same model names—but with built-in analytics and the option to self-host. Migration takes two lines of code.

  ## Quick Migration [#quick-migration]

  Change your base URL and API key:

  ```diff
  - const baseURL = "https://openrouter.ai/api/v1";
  - const apiKey = process.env.OPENROUTER_API_KEY;
  + const baseURL = "https://api.llmgateway.io/v1";
  + const apiKey = process.env.LLM_GATEWAY_API_KEY;
  ```
</section>

<section>
  ## Migration Steps [#migration-steps]

  <Steps>
    <Step>
      ### Get Your LLM Gateway API Key [#get-your-llm-gateway-api-key]

      Sign up at [llmgateway.io/signup](https://llmgateway.io/signup) and create an API key from your dashboard.
    </Step>

    <Step>
      ### Update Environment Variables [#update-environment-variables]

      ```bash
      # Remove OpenRouter credentials
      # OPENROUTER_API_KEY=sk-or-...

      # Add LLM Gateway credentials
      LLM_GATEWAY_API_KEY=llmgtwy_your_key_here
      ```
    </Step>

    <Step>
      ### Update Your Code [#update-your-code]

      #### Using fetch/axios [#using-fetchaxios]

      ```typescript
      // Before (OpenRouter)
      const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
      	method: "POST",
      	headers: {
      		Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
      		"Content-Type": "application/json",
      	},
      	body: JSON.stringify({
      		model: "openai/gpt-5.2",
      		messages: [{ role: "user", content: "Hello!" }],
      	}),
      });

      // After (LLM Gateway)
      const response = await fetch("https://api.llmgateway.io/v1/chat/completions", {
      	method: "POST",
      	headers: {
      		Authorization: `Bearer ${process.env.LLM_GATEWAY_API_KEY}`,
      		"Content-Type": "application/json",
      	},
      	body: JSON.stringify({
      		model: "gpt-5.2",
      		messages: [{ role: "user", content: "Hello!" }],
      	}),
      });
      ```

      #### Using OpenAI SDK [#using-openai-sdk]

      ```typescript
      import OpenAI from "openai";

      // Before (OpenRouter)
      const client = new OpenAI({
      	baseURL: "https://openrouter.ai/api/v1",
      	apiKey: process.env.OPENROUTER_API_KEY,
      });

      // After (LLM Gateway)
      const client = new OpenAI({
      	baseURL: "https://api.llmgateway.io/v1",
      	apiKey: process.env.LLM_GATEWAY_API_KEY,
      });

      // Usage remains the same
      const completion = await client.chat.completions.create({
      	model: "anthropic/claude-3-5-sonnet-20241022",
      	messages: [{ role: "user", content: "Hello!" }],
      });
      ```

      #### Using Vercel AI SDK [#using-vercel-ai-sdk]

      Both OpenRouter and LLM Gateway have native AI SDK providers, making migration straightforward:

      ```typescript
      import { generateText } from "ai";

      // Before (OpenRouter AI SDK Provider)
      import { createOpenRouter } from "@openrouter/ai-sdk-provider";

      const openrouter = createOpenRouter({
      	apiKey: process.env.OPENROUTER_API_KEY,
      });

      const { text } = await generateText({
      	model: openrouter("gpt-5.2"),
      	prompt: "Hello!",
      });

      // After (LLM Gateway AI SDK Provider)
      import { createLLMGateway } from "@llmgateway/ai-sdk-provider";

      const llmgateway = createLLMGateway({
      	apiKey: process.env.LLMGATEWAY_API_KEY,
      });

      const { text } = await generateText({
      	model: llmgateway("gpt-5.2"),
      	prompt: "Hello!",
      });
      ```
    </Step>
  </Steps>
</section>

<section>
  ## Model Name Mapping [#model-name-mapping]

  Most model names are compatible, but here are some common mappings:

  | OpenRouter Model                 | LLM Gateway Model                                                 |
  | -------------------------------- | ----------------------------------------------------------------- |
  | openai/gpt-5.2                   | gpt-5.2 or openai/gpt-5.2                                         |
  | gemini/gemini-3-flash-preview    | gemini-3-flash-preview or google-ai-studio/gemini-3-flash-preview |
  | bedrock/claude-opus-4-5-20251101 | claude-opus-4-5-20251101 or aws-bedrock/claude-opus-4-5-20251101  |

  Check the [models page](https://llmgateway.io/models) for the full list of available models.
</section>

<section>
  ## Streaming Support [#streaming-support]

  LLM Gateway supports streaming responses identically to OpenRouter:

  ```typescript
  const stream = await client.chat.completions.create({
  	model: "anthropic/claude-3-5-sonnet-20241022",
  	messages: [{ role: "user", content: "Write a story" }],
  	stream: true,
  });

  for await (const chunk of stream) {
  	process.stdout.write(chunk.choices[0]?.delta?.content || "");
  }
  ```
</section>

<section>
  ## Full Comparison [#full-comparison]

  Want to see a detailed breakdown of all features? Check out our [LLM Gateway vs OpenRouter comparison page](https://llmgateway.io/compare/open-router).
</section>

<footer>
  ## Need Help? [#need-help]

  * Browse available models at [llmgateway.io/models](https://llmgateway.io/models)
  * Read the [API documentation](https://docs.llmgateway.io)
  * Contact support at [contact@llmgateway.io](mailto:contact@llmgateway.io)
</footer>


# Migrate from Vercel AI Gateway
URL: https://docs.llmgateway.io/migrations/vercel-ai-gateway


<section>

  ## Quick Migration [#quick-migration]

  Swap your provider imports—your AI SDK code stays the same:

  ```diff
  - import { openai } from "@ai-sdk/openai";
  - import { anthropic } from "@ai-sdk/anthropic";
  + import { generateText } from "ai";
  + import { createLLMGateway } from "@llmgateway/ai-sdk-provider";

  + const llmgateway = createLLMGateway({
  +   apiKey: process.env.LLM_GATEWAY_API_KEY
  + });

  const { text } = await generateText({
  -   model: openai("gpt-5.2"),
  +   model: llmgateway("gpt-5.2"),
    prompt: "Hello!"
  });
  ```

  The key difference: one provider, one API key, all models—with caching and analytics built in.
</section>

<section>
  ## Zero-diff alternative: repoint the base URL [#zero-diff-alternative-repoint-the-base-url]

  If your app passes bare model strings (`model: "anthropic/claude-sonnet-5"`), it resolves them through `@ai-sdk/gateway` — the AI SDK's default provider. LLM Gateway implements that protocol, so you can keep every line of application code and repoint the provider instead:

  ```ts
  import { createGateway } from "@ai-sdk/gateway";

  globalThis.AI_SDK_DEFAULT_PROVIDER = createGateway({
  	baseURL: "https://api.llmgateway.io/v4/ai",
  	apiKey: process.env.LLM_GATEWAY_API_KEY,
  });
  ```

  No import changes, no model-string changes, and provider-native web search keeps returning `source-url` parts. See [AI SDK Gateway protocol](https://docs.llmgateway.io/developers/ai-sdk-gateway-protocol).

  Prefer the explicit `@llmgateway/ai-sdk-provider` migration below when you want the gateway's own model IDs and options surfaced as first-class provider settings.
</section>

<section>
  ## Migration Steps [#migration-steps]

  <Steps>
    <Step>
      ### Get Your LLM Gateway API Key [#get-your-llm-gateway-api-key]

      Sign up at [llmgateway.io/signup](https://llmgateway.io/signup) and create an API key from your dashboard.
    </Step>

    <Step>
      ### Install the LLM Gateway AI SDK Provider [#install-the-llm-gateway-ai-sdk-provider]

      Install the native LLM Gateway provider for the Vercel AI SDK:

      ```bash
      pnpm add @llmgateway/ai-sdk-provider
      ```

      This package provides full compatibility with the Vercel AI SDK and supports all LLM Gateway features.
    </Step>

    <Step>
      ### Update Your Code [#update-your-code]

      #### Basic Text Generation [#basic-text-generation]

      ```typescript
      // Before (Vercel AI Gateway with native providers)
      import { openai } from "@ai-sdk/openai";
      import { anthropic } from "@ai-sdk/anthropic";
      import { generateText } from "ai";

      const { text: openaiText } = await generateText({
      	model: openai("gpt-4o"),
      	prompt: "Hello!",
      });

      const { text: claudeText } = await generateText({
      	model: anthropic("claude-3-5-sonnet-20241022"),
      	prompt: "Hello!",
      });

      // After (LLM Gateway - single provider for all models)
      import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
      import { generateText } from "ai";

      const llmgateway = createLLMGateway({
      	apiKey: process.env.LLM_GATEWAY_API_KEY,
      });

      const { text: openaiText } = await generateText({
      	model: llmgateway("openai/gpt-4o"),
      	prompt: "Hello!",
      });

      const { text: claudeText } = await generateText({
      	model: llmgateway("anthropic/claude-3-5-sonnet-20241022"),
      	prompt: "Hello!",
      });
      ```

      #### Streaming Responses [#streaming-responses]

      ```typescript
      import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
      import { streamText } from "ai";

      const llmgateway = createLLMGateway({
      	apiKey: process.env.LLM_GATEWAY_API_KEY,
      });

      const { textStream } = await streamText({
      	model: llmgateway("anthropic/claude-3-5-sonnet-20241022"),
      	prompt: "Write a poem about coding",
      });

      for await (const text of textStream) {
      	process.stdout.write(text);
      }
      ```

      #### Using in Next.js API Routes [#using-in-nextjs-api-routes]

      ```typescript
      // app/api/chat/route.ts
      import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
      import { streamText } from "ai";

      const llmgateway = createLLMGateway({
      	apiKey: process.env.LLM_GATEWAY_API_KEY,
      });

      export async function POST(req: Request) {
      	const { messages } = await req.json();

      	const result = await streamText({
      		model: llmgateway("openai/gpt-4o"),
      		messages,
      	});

      	return result.toDataStreamResponse();
      }
      ```

      #### Alternative: Using OpenAI SDK Adapter [#alternative-using-openai-sdk-adapter]

      If you prefer not to install a new package, you can use `@ai-sdk/openai` with a custom base URL:

      ```typescript
      import { createOpenAI } from "@ai-sdk/openai";
      import { generateText } from "ai";

      const llmgateway = createOpenAI({
      	baseURL: "https://api.llmgateway.io/v1",
      	apiKey: process.env.LLM_GATEWAY_API_KEY,
      });

      const { text } = await generateText({
      	model: llmgateway("openai/gpt-4o"),
      	prompt: "Hello!",
      });
      ```
    </Step>

    <Step>
      ### Update Environment Variables [#update-environment-variables]

      ```bash
      # Remove individual provider keys (optional - can keep as backup)
      # OPENAI_API_KEY=sk-...
      # ANTHROPIC_API_KEY=sk-ant-...

      # Add LLM Gateway key
      export LLM_GATEWAY_API_KEY=llmgtwy_your_key_here
      ```
    </Step>
  </Steps>
</section>

<section>
  ## Model Name Format [#model-name-format]

  LLM Gateway supports two model ID formats:

  **Canonical Model IDs** (without provider prefix) - Uses smart routing to automatically select the best provider based on uptime, throughput, price, and latency:

  ```
  gpt-4o
  claude-3-5-sonnet-20241022
  gemini-1.5-pro
  ```

  **Provider-Prefixed Model IDs** - Routes to a specific provider with automatic failover if uptime drops below 90%:

  ```
  openai/gpt-4o
  anthropic/claude-3-5-sonnet-20241022
  google-ai-studio/gemini-1.5-pro
  ```

  For more details on routing behavior, see the [routing documentation](https://docs.llmgateway.io/features/routing).

  ### Model Mapping Examples [#model-mapping-examples]

  | Vercel AI SDK                             | LLM Gateway                                                                                        |
  | ----------------------------------------- | -------------------------------------------------------------------------------------------------- |
  | `openai("gpt-4o")`                        | `llmgateway("gpt-4o")` or `llmgateway("openai/gpt-4o")`                                            |
  | `anthropic("claude-3-5-sonnet-20241022")` | `llmgateway("claude-3-5-sonnet-20241022")` or `llmgateway("anthropic/claude-3-5-sonnet-20241022")` |
  | `google("gemini-1.5-pro")`                | `llmgateway("gemini-1.5-pro")` or `llmgateway("google-ai-studio/gemini-1.5-pro")`                  |

  Check the [models page](https://llmgateway.io/models) for the full list of available models.
</section>

<section>
  ## Tool Calling [#tool-calling]

  LLM Gateway supports tool calling through the AI SDK:

  ```typescript
  import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
  import { generateText, tool } from "ai";
  import { z } from "zod";

  const llmgateway = createLLMGateway({
  	apiKey: process.env.LLM_GATEWAY_API_KEY,
  });

  const { text, toolResults } = await generateText({
  	model: llmgateway("openai/gpt-4o"),
  	tools: {
  		weather: tool({
  			description: "Get the weather for a location",
  			parameters: z.object({
  				location: z.string(),
  			}),
  			execute: async ({ location }) => {
  				return { temperature: 72, condition: "sunny" };
  			},
  		}),
  	},
  	prompt: "What's the weather in San Francisco?",
  });
  ```
</section>

<section>
  ## Self-Hosting LLM Gateway [#self-hosting-llm-gateway]

  If you prefer self-hosting, LLM Gateway is available under AGPLv3:

  ```bash
  git clone https://github.com/llmgateway/llmgateway
  cd llmgateway
  pnpm install
  pnpm setup
  pnpm dev
  ```

  This gives you the same managed experience with full control over your infrastructure.
</section>

<footer>
  ## Need Help? [#need-help]

  * Browse available models at [llmgateway.io/models](https://llmgateway.io/models)
  * Read the [API documentation](https://docs.llmgateway.io)
  * Contact support at [contact@llmgateway.io](mailto:contact@llmgateway.io)
</footer>


# Error Handling
URL: https://docs.llmgateway.io/resources/error-handling


On the OpenAI-compatible endpoints, LLMGateway returns errors in the same format as the OpenAI API, so existing OpenAI SDKs and tooling can parse gateway errors without changes. This applies to errors forwarded from upstream providers as well as errors raised by the gateway itself (authentication failures, usage limits, validation problems, timeouts, and so on). The Anthropic-compatible Messages endpoint (`/v1/messages`) instead returns Anthropic-native errors — see [Anthropic Endpoint](#anthropic-endpoint) below.

## Error Format [#error-format]

Errors on the OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/embeddings`, `/v1/images`, `/v1/models`, `/v1/moderations`, `/v1/rerank`, `/v1/responses`, `/v1/videos`) use the standard OpenAI error envelope:

```json
{
	"error": {
		"message": "Unauthorized: LLMGateway API key reached its usage limit.",
		"type": "invalid_request_error",
		"param": null,
		"code": "invalid_api_key"
	}
}
```

| Field           | Description                                                                         |
| --------------- | ----------------------------------------------------------------------------------- |
| `error.message` | Human-readable description of what went wrong.                                      |
| `error.type`    | High-level error category (see the table below).                                    |
| `error.param`   | The request parameter that caused the error, or `null` when not parameter-specific. |
| `error.code`    | A more specific machine-readable code, or `null` when no specific code applies.     |

The HTTP status code on the response always matches the error and is the authoritative signal — read it from the response status line rather than the body.

## Status Codes [#status-codes]

The gateway maps HTTP status codes to OpenAI error types and codes as follows:

| Status | `type`                  | `code`                   |
| ------ | ----------------------- | ------------------------ |
| 400    | `invalid_request_error` | *(varies / `null`)*      |
| 401    | `invalid_request_error` | `invalid_api_key`        |
| 402    | `invalid_request_error` | `billing_error`          |
| 403    | `invalid_request_error` | `permission_denied`      |
| 404    | `invalid_request_error` | `not_found`              |
| 408    | `timeout_error`         | `timeout`                |
| 413    | `invalid_request_error` | `request_too_large`      |
| 415    | `invalid_request_error` | `unsupported_media_type` |
| 429    | `rate_limit_error`      | `rate_limit_exceeded`    |
| 499    | `invalid_request_error` | `request_cancelled`      |
| 504    | `timeout_error`         | `timeout`                |
| 529    | `overloaded`            | `overloaded`             |
| 5xx    | `api_error`             | *(`null`)*               |

<Callout type="info">
  Validation errors raised before a request reaches a provider often include a
  more specific `code` and a `param` pointing at the offending field — for
  example `invalid_json`, `model_not_found`, or
  `unsupported_parameter_combination`.
</Callout>

## Streaming Errors [#streaming-errors]

For streaming requests (`"stream": true`), an error that occurs **after** the stream has started is delivered as an SSE `error` event whose payload uses the same `{ "error": { ... } }` envelope. Errors that occur **before** streaming begins (such as authentication failures) are returned as a normal JSON error response with the appropriate status code.

## Anthropic Endpoint [#anthropic-endpoint]

The Anthropic-compatible Messages endpoint (`/v1/messages`) returns errors in Anthropic's native format instead, so the Anthropic SDK can parse them:

```json
{
	"type": "error",
	"error": {
		"type": "authentication_error",
		"message": "Unauthorized: invalid API key."
	}
}
```

## Related [#related]

* [Rate Limits](https://docs.llmgateway.io/resources/rate-limits) — details on `429` responses and rate limit headers.


# Rate Limits
URL: https://docs.llmgateway.io/resources/rate-limits


LLMGateway applies rate limits to ensure fair usage and protect platform stability. Limits are evaluated in a few independent layers:

* **Per-organization endpoint limits** — a requests-per-minute cap on every API endpoint, scoped to your organization.
* **Free model limits** — additional limits specifically for zero-cost models.
* **Provider limits** — upstream limits enforced per provider/model when applicable.

## Per-Organization Endpoint Limits [#per-organization-endpoint-limits]

Every API endpoint is rate limited per organization using a rolling 60-second window. The limit is independent for each endpoint, so traffic to `/v1/chat/completions` does not consume the budget for `/v1/embeddings`.

The default limits (requests per minute, per organization) are:

| Endpoint                                           | Path                       | Requests / min |
| -------------------------------------------------- | -------------------------- | -------------- |
| Chat completions                                   | `/v1/chat/completions`     | 600            |
| Messages (Anthropic)                               | `/v1/messages`             | 600            |
| Responses                                          | `/v1/responses`            | 600            |
| Embeddings                                         | `/v1/embeddings`           | 1200           |
| Moderations                                        | `/v1/moderations`          | 1200           |
| Rerank                                             | `/v1/rerank`               | 1200           |
| Models                                             | `/v1/models`               | 1200           |
| OCR                                                | `/v1/ocr`                  | 300            |
| Images                                             | `/v1/images`               | 300            |
| Speech                                             | `/v1/audio/speech`         | 300            |
| Transcriptions                                     | `/v1/audio/transcriptions` | 300            |
| Videos                                             | `/v1/videos`               | 120            |
| Realtime (mint)                                    | `/v1/realtime`             | 120            |
| [Key and DevPass usage](https://docs.llmgateway.io/developers/devpass-usage) | `/v1/key`                  | 1200           |
| Credits                                            | `/v1/credits`              | 300            |
| AI SDK protocol                                    | `/v*/ai`                   | 600            |

<Callout type="info">
  **[Enterprise](https://llmgateway.io/enterprise) organizations are exempt**
  from these per-organization endpoint limits. [Contact
  us](mailto:contact@llmgateway.io) about enterprise plans.
</Callout>

### Trust Tiers (account age or spend) [#trust-tiers-account-age-or-spend]

For regular (pay-as-you-go) organizations, limits scale with a **trust tier**. An organization qualifies for a tier when its account is old enough, **or** when its lifetime usage spend is high enough **and** the account meets the tier's minimum age. The tier raises the per-endpoint RPM limits, the [concurrent-request ceiling](#concurrent-request-limits), **and** the daily/monthly USD spend caps below.

| Tier | Qualifies (age, or spend + min age) | RPM multiplier | Concurrent | Daily cap | Monthly cap |
| ---- | ----------------------------------- | -------------- | ---------- | --------- | ----------- |
| 0    | new / $0                            | 1×             | 100        | $25       | $250        |
| 1    | 7 days **or** $10 (account ≥ 1 day) | 2×             | 200        | $100      | $1,000      |
| 2    | 30 days **or** $100 (≥ 3 days)      | 4×             | 400        | $500      | $5,000      |
| 3    | 60 days **or** $1,000 (≥ 7 days)    | 10×            | 1,000      | $5,000    | $50,000     |
| 4    | 90 days **or** $5,000 (≥ 14 days)   | 20×            | 2,000      | $15,000   | $200,000    |

Spend alone never promotes a brand-new account: each spend-qualified tier also requires the minimum account age shown, so the fastest possible path to Tier 4 is 14 days — no amount of day-one usage unlocks higher limits.

For example, an org past 30 days old (or with $100+ of usage) is Tier 2: chat completions rises from 600 to 2,400 RPM, with a $500/day and $5,000/month spend ceiling.

Qualifying spend counts **usage billed to your credit balance only** — usage served through your own provider keys (BYOK) does not count — and is **net of refunds**: every refunded payment is deducted, so refunded or clawed-back money never raises limits. Refunded top-ups still count against the top-up allowance below — refunding does not free up top-up headroom.

### Daily & Monthly Spend Caps [#daily--monthly-spend-caps]

Regular organizations also have hard **USD spend ceilings** — a daily and a monthly cap set by the trust tier above — so a brand-new account has a tight dollar velocity limit that rises as it ages or spends. Only real paid usage counts; **free models are exempt**, as are **enterprise** (no caps) and dev/chat plan orgs (which have their own plan limits). When a cap is reached, requests return `429` until the counter resets (UTC midnight for daily, first of the month for monthly).

### Top-Up Limits [#top-up-limits]

Credit top-ups are also velocity-limited by trust tier: each organization can add at most a tier-scaled gross USD amount to its balance per **rolling 24-hour window**. This applies before any charge is made — a top-up attempt over the allowance is rejected with `429` and no card is charged.

| Tier | Top-up allowance (rolling 24h) |
| ---- | ------------------------------ |
| 0    | $100                           |
| 1    | $500                           |
| 2    | $2,500                         |
| 3    | $10,000                        |
| 4    | $20,000                        |

The limit covers dashboard top-ups (card and hosted checkout), auto top-up, and Dev plan pay-as-you-go top-ups. **[Enterprise](https://llmgateway.io/enterprise) organizations are exempt** — [contact us](mailto:contact@llmgateway.io) if you need a higher allowance. Your current allowance and usage are shown on the Settings → Limits page. Hosted checkout links expire after 30 minutes.

### Dev and Chat Plans [#dev-and-chat-plans]

Organizations on a **Dev plan** (also called devpass) or a **Chat plan** have their own, tighter per-endpoint limits and do not receive the spend-based multiplier.

Dev plans are inference-only and only cover **chat completions, messages, responses, the models list, and OCR**, each at a flat 120 requests-per-minute floor. The other endpoints (embeddings, moderations, images, speech, videos) are **not available*&#x2A; on Dev plans and are marked &#x2A;*—** below.

| Endpoint             | Dev plan / min | Chat plan / min |
| -------------------- | -------------- | --------------- |
| Chat completions     | 120            | 60              |
| Messages (Anthropic) | 120            | 60              |
| Responses            | 120            | 60              |
| Models               | 120            | 120             |
| OCR                  | 120            | 30              |
| Embeddings           | —              | 120             |
| Moderations          | —              | 120             |
| Images               | —              | 30              |
| Speech               | —              | 30              |
| Videos               | —              | 12              |

### Enterprise [#enterprise]

Organizations on the **[Enterprise plan](https://llmgateway.io/enterprise) have no per-organization requests-per-minute, spend, or top-up limits**. Your request rate is limited only by your credit balance and any upstream provider limits. The only gateway limit that still applies is a greatly elevated [concurrent-request ceiling](#concurrent-request-limits).

<Callout type="info">
  Need unlimited gateway throughput? [Contact us](mailto:contact@llmgateway.io)
  about an enterprise plan.
</Callout>

## Concurrent Request Limits [#concurrent-request-limits]

Separately from the per-minute request limits above, each organization has one fleet-wide budget of **concurrent in-flight requests** across all inference endpoints (chat completions, messages, responses, embeddings, moderations, rerank, OCR, images, speech, transcriptions, videos, and the AI SDK surface). A slot is held for a request's full lifetime — including the entire duration of a streamed response — and freed when the response finishes or the connection closes.

This bounds what a per-minute limit cannot: long-running requests. Six hundred requests per minute that each stream for two minutes hold 1,200 connections open; the concurrency budget is what keeps that pile-up from exhausting shared gateway capacity.

For regular (pay-as-you-go) organizations the ceiling scales with the same [trust tier](#trust-tiers-account-age-or-spend) that raises the per-minute limits; Dev and Chat plans have a flat limit.

| Plan                    | Concurrent requests |
| ----------------------- | ------------------- |
| Regular (PAYG) — Tier 0 | 100                 |
| Regular (PAYG) — Tier 1 | 200                 |
| Regular (PAYG) — Tier 2 | 400                 |
| Regular (PAYG) — Tier 3 | 1,000               |
| Regular (PAYG) — Tier 4 | 2,000               |
| Dev plan                | 50                  |
| Chat plan               | 10                  |
| Enterprise              | 2,000               |

Unlike the per-minute limits, **Enterprise organizations are not exempt** — they get the elevated ceiling instead. Requests over the limit receive a retryable `429`:

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 1
```

```json
{
	"error": {
		"message": "Too many concurrent requests for this organization (limit: 100). Retry shortly, or reduce request concurrency.",
		"type": "rate_limit_error",
		"code": "rate_limit_exceeded"
	}
}
```

Because slots free up continuously as in-flight requests complete, retrying after a short backoff typically succeeds — there is no fixed window to wait out. If you consistently hit the concurrency limit, reduce your client-side parallelism or [contact us](mailto:contact@llmgateway.io) about raising your ceiling.

## Free Models [#free-models]

Free models (models with zero input and output pricing) have additional rate limits that depend on your account's credit status:

### Base Rate Limits [#base-rate-limits]

For organizations with **zero credits**:

* **5 requests per 10 minutes**
* Applies to all free model requests
* Resets every 10 minutes

### Elevated Rate Limits [#elevated-rate-limits]

For organizations that have **purchased at least some credits**:

* **20 requests per minute**
* Applies to all free model requests
* Resets every minute

<Callout type="info">
  When using free models with elevated limits, your credits will **not** be
  deducted. The elevated rate limits are simply a benefit for users who have
  added credits to their account.
</Callout>

## Provider Limits [#provider-limits]

Some providers and models enforce their own upstream request limits. When a provider limit is reached, LLMGateway returns a `429` and includes provider-scoped headers (`X-RateLimit-Limit-Provider`, `X-RateLimit-Remaining-Provider`). Where possible, [fallback routing](https://docs.llmgateway.io/features/routing) automatically retries the request on a healthy provider.

## Rate Limit Headers [#rate-limit-headers]

When a request is rate limited, the `429` response includes:

```http
Retry-After: 12
X-RateLimit-Limit: 600
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1640995200
```

* `Retry-After`: Seconds to wait before retrying
* `X-RateLimit-Limit`: Maximum number of requests allowed in the current window
* `X-RateLimit-Remaining`: Number of requests remaining in the current window
* `X-RateLimit-Reset`: Unix timestamp when the rate limit window resets

## Rate Limit Exceeded [#rate-limit-exceeded]

When you exceed a rate limit, you'll receive a `429 Too Many Requests` response:

```json
{
	"error": {
		"message": "Rate limit exceeded for /v1/chat/completions. Please retry after 12 seconds.",
		"type": "rate_limit_error",
		"code": "rate_limit_exceeded"
	}
}
```

This uses the standard OpenAI-compatible error envelope. Requests to the Anthropic-compatible `/v1/messages` endpoint receive the Anthropic error shape instead. See [Error Handling](https://docs.llmgateway.io/resources/error-handling) for the full format and status-code reference.

## Gateway Overload (529) [#gateway-overload-529]

Separately from per-account rate limits, the gateway protects itself from
transient overload. When a single gateway instance is holding too many
concurrent in-flight inference requests at once — across all organizations
combined (for example during a traffic spike, or when an upstream provider is
slow and connections pile up) — it sheds excess inference requests with an
`HTTP 529` response instead of letting them queue indefinitely. Non-inference
endpoints such as the models list are unaffected:

```http
HTTP/1.1 529
Retry-After: 1
```

```json
{
	"error": {
		"message": "Gateway overloaded, please retry",
		"type": "overloaded",
		"code": "overloaded"
	}
}
```

Requests to the Anthropic-compatible `/v1/messages` endpoint receive the
equivalent Anthropic envelope (`{ "type": "error", "error": { "type": "overloaded_error" } }`),
matching Anthropic's own `529` behavior.

<Callout type="info">
  A `529` is **transient and retryable** — it reflects momentary capacity, not a
  quota on your account. Unlike a `429`, it is not tied to your credits or model
  tier, and retrying after a short delay (honoring the `Retry-After` header)
  will typically succeed.
</Callout>

How `529` differs from `429`:

|           | `429 Too Many Requests`                                                                        | `529` Overloaded                       |
| --------- | ---------------------------------------------------------------------------------------------- | -------------------------------------- |
| Cause     | Your organization exceeded its request rate or [concurrency](#concurrent-request-limits) limit | The gateway is momentarily at capacity |
| Scope     | Per organization / API key                                                                     | Transient, server-side                 |
| Fix       | Slow down or reduce concurrency; add credits for [elevated limits](#elevated-rate-limits)      | Retry after a short delay              |
| Retryable | After the window resets (rate) or as soon as an in-flight request finishes (concurrency)       | Yes, immediately with backoff          |

## Best Practices [#best-practices]

* **Respect `Retry-After`.** Implement exponential backoff when you receive `429` or `529` responses, starting from the `Retry-After` value.
* **Watch the headers.** Monitor `X-RateLimit-Remaining` to back off before you hit the limit.
* **Spread traffic across endpoints.** Limits are per endpoint, so unrelated workloads don't compete for the same budget.
* **Scale with usage.** Regular organizations unlock higher limits automatically as lifetime spend grows; [contact us](mailto:contact@llmgateway.io) about an [Enterprise plan](https://llmgateway.io/enterprise) to remove the per-minute limits and get an elevated concurrency ceiling.

<Callout type="success">
  Adding even a small amount of credits to your account (e.g., $10) will
  immediately upgrade your free model rate limits from 5 requests per 10 minutes
  to 20 requests per minute.
</Callout>


# AWS
URL: https://docs.llmgateway.io/self-host/aws


<Callout type="info">
  **Deploy in one command.** Our [Enterprise
  plan](https://llmgateway.io/enterprise) includes Terraform modules that
  provision the cluster, database, cache, networking, and secrets and deploy LLM
  Gateway on AWS — no manual setup required. [Learn
  more](https://llmgateway.io/enterprise).
</Callout>

This guide covers a production deployment of LLM Gateway on AWS using EKS for the application services and managed AWS services for the backing stores.

## Architecture [#architecture]

| Component  | AWS service                  |
| ---------- | ---------------------------- |
| Compute    | Amazon EKS (Kubernetes)      |
| PostgreSQL | Amazon RDS for PostgreSQL    |
| Redis      | Amazon ElastiCache for Redis |
| Secrets    | AWS Secrets Manager          |
| Ingress    | AWS Load Balancer Controller |

## What to configure [#what-to-configure]

### 1. PostgreSQL — Amazon RDS [#1-postgresql--amazon-rds]

Create an RDS for PostgreSQL instance in a private subnet. Enable automated backups and Multi-AZ for high availability. Note the connection string for `DATABASE_URL`.

### 2. Redis — Amazon ElastiCache [#2-redis--amazon-elasticache]

Create an ElastiCache for Redis cluster in the same VPC. Use it for `REDIS_URL`. A single-node cluster is fine to start; enable replication for production.

### 3. Compute — Amazon EKS [#3-compute--amazon-eks]

Create an EKS cluster with a managed node group sized for your traffic. Install the [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/) to expose the gateway through an Application Load Balancer.

### 4. Networking [#4-networking]

Run RDS and ElastiCache in private subnets and allow inbound traffic only from the EKS node security group. Expose only the gateway (and optionally the UI) to the internet through the load balancer.

### 5. Secrets — AWS Secrets Manager [#5-secrets--aws-secrets-manager]

Store `AUTH_SECRET`, `GATEWAY_API_KEY_HASH_SECRET`, and your provider API keys in Secrets Manager. Sync them into the cluster with the [External Secrets Operator](https://external-secrets.io/) or the AWS Secrets and Configuration Provider (ASCP).

## Deploy the Helm chart [#deploy-the-helm-chart]

With the backing services in place, deploy LLM Gateway with the [Helm chart](https://docs.llmgateway.io/self-host/kubernetes), pointing it at your RDS and ElastiCache endpoints:

```bash
helm install llmgateway oci://ghcr.io/theopenco/charts/llmgateway -f values.yaml
```

```yaml
config:
  DATABASE_URL: "postgres://user:password@your-rds-endpoint:5432/llmgateway"
  REDIS_URL: "redis://your-elasticache-endpoint:6379"
  AUTH_SECRET: "from-secrets-manager"
  GATEWAY_API_KEY_HASH_SECRET: "from-secrets-manager"
```

See the [Kubernetes guide](https://docs.llmgateway.io/self-host/kubernetes) for the full set of configurable values and how to scale the gateway.

<Callout type="info">
  **Prefer not to wire this up by hand?** The [Enterprise
  plan](https://llmgateway.io/enterprise) ships Terraform modules that stand up
  the entire AWS stack — EKS, RDS, ElastiCache, networking, and secrets — and
  deploy LLM Gateway in one command. [Talk to
  us](https://llmgateway.io/enterprise#contact).
</Callout>


# Azure
URL: https://docs.llmgateway.io/self-host/azure


<Callout type="info">
  **Deploy in one command.** Our [Enterprise
  plan](https://llmgateway.io/enterprise) includes Terraform modules that
  provision the cluster, database, cache, networking, and secrets and deploy LLM
  Gateway on Azure — no manual setup required. [Learn
  more](https://llmgateway.io/enterprise).
</Callout>

This guide covers a production deployment of LLM Gateway on Azure using AKS for the application services and managed Azure services for the backing stores.

## Architecture [#architecture]

| Component  | Azure service                     |
| ---------- | --------------------------------- |
| Compute    | Azure Kubernetes Service (AKS)    |
| PostgreSQL | Azure Database for PostgreSQL     |
| Redis      | Azure Cache for Redis             |
| Secrets    | Azure Key Vault                   |
| Ingress    | Application Gateway / AKS Ingress |

## What to configure [#what-to-configure]

### 1. PostgreSQL — Azure Database for PostgreSQL [#1-postgresql--azure-database-for-postgresql]

Create an Azure Database for PostgreSQL Flexible Server with private access. Enable automated backups and zone-redundant high availability. Note the connection details for `DATABASE_URL`.

### 2. Redis — Azure Cache for Redis [#2-redis--azure-cache-for-redis]

Create an Azure Cache for Redis instance in the same virtual network and use its endpoint for `REDIS_URL`. Choose a Standard or Premium tier for replication in production.

### 3. Compute — AKS [#3-compute--aks]

Create an AKS cluster with a node pool sized for your traffic. Use the [Application Gateway Ingress Controller](https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-overview) or an NGINX ingress to expose the gateway.

### 4. Networking [#4-networking]

Deploy the database and cache with private endpoints inside your virtual network and restrict access to the AKS subnet. Expose only the gateway (and optionally the UI) to the internet.

### 5. Secrets — Azure Key Vault [#5-secrets--azure-key-vault]

Store `AUTH_SECRET`, `GATEWAY_API_KEY_HASH_SECRET`, and your provider API keys in Key Vault. Sync them into the cluster with the [Azure Key Vault Provider for Secrets Store CSI Driver](https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver).

## Deploy the Helm chart [#deploy-the-helm-chart]

With the backing services in place, deploy LLM Gateway with the [Helm chart](https://docs.llmgateway.io/self-host/kubernetes), pointing it at your Azure Database and Azure Cache endpoints:

```bash
helm install llmgateway oci://ghcr.io/theopenco/charts/llmgateway -f values.yaml
```

```yaml
config:
  DATABASE_URL: "postgres://user:password@your-postgres-host:5432/llmgateway"
  REDIS_URL: "redis://your-redis-host:6380"
  AUTH_SECRET: "from-key-vault"
  GATEWAY_API_KEY_HASH_SECRET: "from-key-vault"
```

See the [Kubernetes guide](https://docs.llmgateway.io/self-host/kubernetes) for the full set of configurable values and how to scale the gateway.

<Callout type="info">
  **Prefer not to wire this up by hand?** The [Enterprise
  plan](https://llmgateway.io/enterprise) ships Terraform modules that stand up
  the entire Azure stack — AKS, Azure Database for PostgreSQL, Azure Cache for
  Redis, networking, and secrets — and deploy LLM Gateway in one command. [Talk
  to us](https://llmgateway.io/enterprise#contact).
</Callout>


# Docker Compose
URL: https://docs.llmgateway.io/self-host/docker-compose


Docker Compose runs each service in its own container, giving you more control over scaling and configuration than the [single Docker image](https://docs.llmgateway.io/self-host/docker). It's a good fit for a single production host. For multi-node, high-availability deployments, use [Kubernetes](https://docs.llmgateway.io/self-host/kubernetes).

## Prerequisites [#prerequisites]

* Latest Docker with Compose
* API keys for the LLM providers you want to use (OpenAI, Anthropic, etc.)

## Option A: Unified image with Compose [#option-a-unified-image-with-compose]

Run the all-in-one image under Compose for easy lifecycle management:

```bash
# Download the compose file
curl -O https://raw.githubusercontent.com/theopenco/llmgateway/main/infra/docker-compose.unified.yml
curl -O https://raw.githubusercontent.com/theopenco/llmgateway/main/.env.unified.example

# Configure environment
cp .env.unified.example .env
# Edit .env with your configuration

# Start the service
docker compose -f docker-compose.unified.yml up -d
```

## Option B: Separate services [#option-b-separate-services]

Run each service in its own container for the most flexibility:

```bash
# Clone the repository
git clone https://github.com/theopenco/llmgateway.git
cd llmgateway

# Configure environment
cp .env.example .env
# Edit .env with your configuration

# Start the services
docker compose -f infra/docker-compose.split.yml up -d
```

Pin to a specific version by replacing the `latest` image tags with the [latest release](https://github.com/theopenco/llmgateway/releases). To build the images from source, use the `*.local.yml` compose files in the `infra` directory.

## Accessing your instance [#accessing-your-instance]

* **Web Interface**: [http://localhost:3002](http://localhost:3002)
* **Chat Playground**: [http://localhost:3003](http://localhost:3003)
* **Documentation**: [http://localhost:3005](http://localhost:3005)
* **Provider Portal (Airside)**: [http://localhost:3007](http://localhost:3007)
* **API Endpoint**: [http://localhost:4002](http://localhost:4002)
* **Gateway Endpoint**: [http://localhost:4001](http://localhost:4001)

## Required configuration [#required-configuration]

At minimum, set these environment variables:

```bash
# Database (change the password!)
POSTGRES_PASSWORD=your_secure_password_here

# Authentication
AUTH_SECRET=your-secret-key-here
GATEWAY_API_KEY_HASH_SECRET=your-api-key-hash-secret-here

# LLM Provider API Keys (add the ones you need)
LLM_OPENAI_API_KEY=sk-...
LLM_ANTHROPIC_API_KEY=sk-ant-...
```

## Enterprise license [#enterprise-license]

Add the signed license supplied with your Enterprise agreement to `.env`:

```bash
LLMGATEWAY_ENTERPRISE_LICENSE=your-signed-license
```

Restart the API and gateway containers after changing it. Every replica may use
the same license. The token is verified offline against a public key included in
the images; it is never sent to LLM Gateway. Core features continue working when
the key is missing or invalid. A standard Enterprise license is bound to one
organization. A white-label license permits multiple Enterprise organizations
and unlocks the multi-organization admin dashboard. Expired licenses receive a
seven-day grace period.

After restart, a signed-in administrator can check `GET /admin/license` for the
state, expiry, scope, seat cap, and current distinct-seat usage. For signing-key
rotation, upgrade the API and gateway images so they contain the new public key
before installing a token with the new key ID. Old public keys remain in the
image during the rotation window.

See [Enterprise licensing](https://docs.llmgateway.io/self-host/enterprise-license) for license types,
installation, verification, expiry behavior, and key rotation.

## Management commands [#management-commands]

```bash
# View logs
docker compose -f infra/docker-compose.split.yml logs -f

# Restart services
docker compose -f infra/docker-compose.split.yml restart

# Stop services
docker compose -f infra/docker-compose.split.yml down
```

## Managing provider credentials from the admin dashboard [#managing-provider-credentials-from-the-admin-dashboard]

Provider credentials can also live in the database instead of the environment.
The admin dashboard's **Provider Credentials** page lets you add, edit and remove
them without redeploying, and each credential carries every setting the provider
needs — API key, base URL, project, region, resource, API version — plus a free-form
note so several keys for the same provider stay tellable apart.

Resolution is per provider: as soon as a provider has at least one managed
credential, credits-mode requests to it use those credentials and its `LLM_*`
environment variables are ignored entirely. Providers with no managed credential
keep reading the environment, so you can migrate one provider at a time.

Managed credentials cover the same axes as the environment variables they replace:

* Several credentials per provider, selected with the same health-aware routing
  as the comma-separated env lists.
* A per-credential audience (all organizations / enterprise plans / DevPass and
  Chat plans), mirroring the `__ENTERPRISE` and `__PLANS` suffixes below.
* An optional region, mirroring the `{ENV_VAR}__{REGION}` overrides.

A request that resolves no region is served from the provider's default region,
so it is served by a region-agnostic credential or by one pinned to that default
region. Providers whose keys are region-scoped — those with no global region, so
every credential belongs to exactly one region — are therefore fully covered by
one credential per region, with no region-agnostic credential needed. Providers
whose key works across every region (AWS Bedrock) are stricter: a region on the
credential scopes it to that region only, and a request for another region fails
rather than borrowing it.

Video generation pins the credential that created a job onto the job itself, so
polling and content retrieval hours later go back through the same credential
rather than re-selecting one.

Provider credential tokens are encrypted at rest with AES-256-GCM because the
gateway must recover them for upstream requests. New and rolled gateway API
keys are stored only as keyed HMAC-SHA-256 fingerprints; incoming secrets are
fingerprinted and compared during authentication. Both protections derive from
`GATEWAY_API_KEY_HASH_SECRET`, so there is no extra variable to set. Give it a
strong random value (`openssl rand -base64 32`) and configure that same value
on every service.

For rotation, prepend the same new secret to the comma-separated keyring on
every service and retain the old entries while credentials and hashes still use
them. Authentication checks every retained entry, while new hashes use the
first entry. Removing an old entry makes its provider credentials undecryptable
and its remaining hashes unusable.

## Multiple API keys and load balancing [#multiple-api-keys-and-load-balancing]

LLM Gateway supports multiple API keys per provider for load balancing and increased availability. Provide comma-separated values:

```bash
# Multiple OpenAI keys for load balancing
LLM_OPENAI_API_KEY=sk-key1,sk-key2,sk-key3

# Multiple Anthropic keys
LLM_ANTHROPIC_API_KEY=sk-ant-key1,sk-ant-key2
```

### Health-aware routing [#health-aware-routing]

The gateway tracks the health of each API key and routes requests to healthy keys. If a key returns consecutive errors, it's temporarily skipped. Keys that return authentication errors (401/403) are blacklisted until restart.

### Related configuration values [#related-configuration-values]

For providers that require additional configuration (like Google Vertex), specify multiple values that correspond to each API key. The gateway uses the matching index:

```bash
# Multiple Google Vertex configurations
LLM_GOOGLE_VERTEX_API_KEY=key1,key2,key3
LLM_GOOGLE_CLOUD_PROJECT=project-a,project-b,project-c
LLM_GOOGLE_VERTEX_REGION=us-central1,europe-west1,asia-east1
```

When the gateway selects `key2`, it automatically uses `project-b` and `europe-west1`. If you have fewer configuration values than keys, the last value is reused for the remaining keys.

`LLM_GOOGLE_CLOUD_PROJECT` is optional for Google Vertex API-key chat, embedding, and speech requests, which use the projectless publisher-model endpoint when it is unset. Set it for OAuth authentication, video generation, or project-scoped Vertex URLs.

## Enterprise and plan provider env overrides [#enterprise-and-plan-provider-env-overrides]

Any provider env var — API keys, base URLs, regions, Google Cloud projects, Azure resources, and other provider-specific settings — supports optional per-audience overrides:

* `__ENTERPRISE` suffix: used instead of the base var for organizations on the enterprise plan
* `__PLANS` suffix: used instead of the base var for plan-based (non-PAYG) organizations — DevPass coding plans and Chat plans

```bash
# Shared key for all organizations
LLM_OPENAI_API_KEY=sk-shared-key

# Used instead of the shared key, but only for enterprise-plan organizations
LLM_OPENAI_API_KEY__ENTERPRISE=sk-enterprise-key

# Used instead of the shared key, but only for plan-based (DevPass/Chat plan)
# organizations
LLM_OPENAI_API_KEY__PLANS=sk-plans-key

# Companion settings can be overridden the same way, e.g. a dedicated
# Google Cloud project and base URL per audience
LLM_GOOGLE_CLOUD_PROJECT__ENTERPRISE=enterprise-gcp-project
LLM_OPENAI_BASE_URL__PLANS=https://plans-proxy.internal
```

Notes:

* Overrides are optional — matching organizations fall back to the base var when an override is unset. All other organizations never read the override vars.
* The base API key is still what makes a provider available for routing, so set it even when you route all enterprise or plan traffic through dedicated keys.
* Comma-separated values, load balancing, and health-aware routing work exactly like the base key; each override key list gets its own independent health tracking.
* A set override replaces the base var wholesale, including its comma-separated list. Indexed companion values (like `LLM_GOOGLE_CLOUD_PROJECT`) resolve at the selected key index from the override list when one is set, otherwise from the base list — so keep an override key list index-compatible with its companion lists (or use single values).
* Region-specific overrides compose with the variant suffix. For matching organizations the API-key lookup order is `{BASE}__ENTERPRISE__{REGION}` → `{BASE}__{REGION}` → `{BASE}__ENTERPRISE` → `{BASE}` (same pattern with `__PLANS`).
* If an organization is both on the enterprise plan and a plan-based org, the enterprise overrides win.


# Docker
URL: https://docs.llmgateway.io/self-host/docker


The unified Docker image bundles every service — UI, API, Gateway, PostgreSQL, and Redis — into a single container. It's the fastest way to get a working instance and is ideal for trying LLM Gateway out or running a single low-traffic deployment.

For production, run each service separately with [Docker Compose](https://docs.llmgateway.io/self-host/docker-compose) or deploy to [Kubernetes](https://docs.llmgateway.io/self-host/kubernetes) with managed Postgres and Redis.

## Prerequisites [#prerequisites]

* Latest Docker
* API keys for the LLM providers you want to use (OpenAI, Anthropic, etc.)

## Run the container [#run-the-container]

```bash
# Set a strong secret first
export LLM_GATEWAY_SECRET="your-secret-key-here"
export GATEWAY_API_KEY_HASH_SECRET="your-api-key-hash-secret-here"
# Required only for licensed Enterprise features
export LLMGATEWAY_ENTERPRISE_LICENSE="your-signed-license"

# Run the container
docker run -d \
  --name llmgateway \
  --restart unless-stopped \
  -p 3002:3002 \
  -p 3003:3003 \
  -p 3005:3005 \
  -p 3006:3006 \
  -p 3007:3007 \
  -p 4001:4001 \
  -p 4002:4002 \
  -v llmgateway_postgres:/var/lib/postgresql/data \
  -v llmgateway_redis:/var/lib/redis \
  -e AUTH_SECRET="$LLM_GATEWAY_SECRET" \
  -e GATEWAY_API_KEY_HASH_SECRET="$GATEWAY_API_KEY_HASH_SECRET" \
  -e LLMGATEWAY_ENTERPRISE_LICENSE="$LLMGATEWAY_ENTERPRISE_LICENSE" \
  ghcr.io/theopenco/llmgateway-unified:latest
```

Docker creates the named volumes automatically on first run. Do not bind-mount a host directory directly to `/var/lib/postgresql/data`, because PostgreSQL initialization inside the container needs to manage permissions on that path.

Pin to a specific version instead of `latest` using the [latest release tag](https://github.com/theopenco/llmgateway/releases).

## Accessing your instance [#accessing-your-instance]

* **Web Interface**: [http://localhost:3002](http://localhost:3002)
* **Chat Playground**: [http://localhost:3003](http://localhost:3003)
* **Documentation**: [http://localhost:3005](http://localhost:3005)
* **Provider Portal (Airside)**: [http://localhost:3007](http://localhost:3007)
* **API Endpoint**: [http://localhost:4002](http://localhost:4002)
* **Gateway Endpoint**: [http://localhost:4001](http://localhost:4001)

## Management commands [#management-commands]

```bash
# View logs
docker logs llmgateway

# Restart the container
docker restart llmgateway

# Stop the container
docker stop llmgateway
```

## Required configuration [#required-configuration]

At minimum, set these environment variables:

```bash
# Authentication
AUTH_SECRET=your-secret-key-here
GATEWAY_API_KEY_HASH_SECRET=your-api-key-hash-secret-here

# LLM Provider API Keys (add the ones you need)
LLM_OPENAI_API_KEY=sk-...
LLM_ANTHROPIC_API_KEY=sk-ant-...
```

## Enterprise license [#enterprise-license]

Core gateway and management functionality works without a license. Enterprise
features require a signed license in production. A standard license is bound to
one organization. A white-label license permits multiple Enterprise
organizations and unlocks the multi-organization admin dashboard. Set
`LLMGATEWAY_ENTERPRISE_LICENSE` and restart the container. Validation is local:
the image contains the public verification key and never sends the license to a
remote service.

An expired license has a seven-day grace period. After grace, Enterprise
features lock while core gateway traffic continues. Contact
[sales](https://llmgateway.io/enterprise) to obtain or renew a license.

See [Enterprise licensing](https://docs.llmgateway.io/self-host/enterprise-license) for license types,
installation, verification, expiry behavior, and key rotation.

LLM Gateway supports multiple API keys per provider for load balancing — provide comma-separated values. See [Docker Compose](https://docs.llmgateway.io/self-host/docker-compose#multiple-api-keys-and-load-balancing) for details. Provider env vars also support optional `__ENTERPRISE` / `__PLANS` suffix overrides used only for enterprise-plan or plan-based (DevPass/Chat plan) organizations — see [Enterprise and plan provider env overrides](https://docs.llmgateway.io/self-host/docker-compose#enterprise-and-plan-provider-env-overrides).

## Next steps [#next-steps]

Once your instance is running:

1. **Open the web interface** at [http://localhost:3002](http://localhost:3002)
2. **Create your first organization** and project
3. **Generate API keys** for your applications
4. **Test the gateway** by making API calls to [http://localhost:4001](http://localhost:4001)


# Enterprise Licensing
URL: https://docs.llmgateway.io/self-host/enterprise-license


LLM Gateway runs without a license: core gateway and management functionality
remain available. A signed license unlocks Enterprise features in production.
Licenses are verified offline using public keys included in the official images;
the token is never sent to LLM Gateway.

To obtain or renew a license, [contact sales](https://llmgateway.io/enterprise).

## Choose a license [#choose-a-license]

| License         | Organization access                   | Admin dashboard |
| --------------- | ------------------------------------- | --------------- |
| **Enterprise**  | One organization bound to the license | No              |
| **White-label** | Multiple Enterprise organizations     | Yes             |

Both license types include a maximum number of distinct users across unlocked
Enterprise organizations. Existing memberships are not removed if usage exceeds
a renewed license's lower limit, but new licensed users are blocked until usage
is below the limit.

For a standard Enterprise license, sales will ask for the organization ID to
bind. It is the first ID after `/dashboard/` in that organization's dashboard
URL.

Installing a license does not change an organization's recorded plan. The bound
organization must also be provisioned on the Enterprise plan; confirm that step
with sales when the license is issued. White-label operators can assign the
Enterprise plan from the multi-organization admin dashboard.

## Docker Compose [#docker-compose]

Add the signed token supplied with your agreement to the `.env` file beside your
Compose file:

```bash
LLMGATEWAY_ENTERPRISE_LICENSE=your-signed-license
```

Keep `.env` out of version control. The provided split and unified Compose files
pass this value to the processes that verify entitlements.

Recreate the affected containers so they receive the new environment value:

```bash
# Split deployment
docker compose -f infra/docker-compose.split.yml up -d --force-recreate api gateway

# Unified deployment
docker compose -f docker-compose.unified.yml up -d --force-recreate llmgateway
```

A plain `docker compose restart` does not reload changed environment variables.

## Docker [#docker]

Add this option to the complete `docker run` command in the
[Docker guide](https://docs.llmgateway.io/self-host/docker):

```bash
-e LLMGATEWAY_ENTERPRISE_LICENSE="your-signed-license"
```

When replacing a license, recreate the container with the new value.

## Kubernetes and Helm [#kubernetes-and-helm]

Store the token under the `LLMGATEWAY_ENTERPRISE_LICENSE` key in the Kubernetes
Secret referenced by the chart's `existingSecret` value:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: llmgateway-secrets
type: Opaque
stringData:
  LLMGATEWAY_ENTERPRISE_LICENSE: "your-signed-license"
  # Include the chart's other required secrets here.
```

```yaml
existingSecret: llmgateway-secrets
```

The existing Secret must also contain the other values required by the chart.
Restart the API and gateway workloads after rotating the Secret. For a temporary
or non-production installation, the chart can create the Secret from
`enterprise.license`, but do not commit a token to a values file.

## Verify the license [#verify-the-license]

Sign in as a deployment administrator and request `GET /admin/license` from the
internal API. The response includes the license state, kind, expiration, seat
limit, distinct seats used, and remaining seats. It never returns the token or
customer identifier.

The possible states are:

| State                                               | Enterprise access                                        |
| --------------------------------------------------- | -------------------------------------------------------- |
| `active`                                            | Enabled                                                  |
| `grace`                                             | Enabled with an expiry warning                           |
| `missing`, `invalid`, `not_yet_valid`, or `expired` | Locked; core functionality remains available             |
| `development`                                       | Enabled only when running from source outside production |

The seven-day grace period begins exactly when the license expires. Enterprise
features lock at the end of grace, but health checks and core gateway traffic
continue working.

## Rotate licenses and verification keys [#rotate-licenses-and-verification-keys]

Replacing a token with another token signed by an already-supported key only
requires updating the deployment secret and recreating or restarting the API and
gateway processes.

When a token uses a new key ID, first upgrade the API and gateway images to a
version containing its public verification key. Then install the new token. Old
public keys remain in images during the rotation window, so replicas can be
upgraded before the license changes.

Treat the signed token like a password: store it in your secret manager, do not
commit it, and do not include it in logs or support messages.


# Google Cloud
URL: https://docs.llmgateway.io/self-host/gcp


<Callout type="info">
  **Deploy in one command.** Our [Enterprise
  plan](https://llmgateway.io/enterprise) includes Terraform modules that
  provision the cluster, database, cache, networking, and secrets and deploy LLM
  Gateway on Google Cloud — no manual setup required. [Learn
  more](https://llmgateway.io/enterprise).
</Callout>

This guide covers a production deployment of LLM Gateway on Google Cloud using GKE for the application services and managed Google Cloud services for the backing stores.

## Architecture [#architecture]

| Component  | Google Cloud service               |
| ---------- | ---------------------------------- |
| Compute    | Google Kubernetes Engine (GKE)     |
| PostgreSQL | Cloud SQL for PostgreSQL           |
| Redis      | Memorystore for Redis              |
| Secrets    | Secret Manager                     |
| Ingress    | GKE Ingress / Cloud Load Balancing |

## What to configure [#what-to-configure]

### 1. PostgreSQL — Cloud SQL [#1-postgresql--cloud-sql]

Create a Cloud SQL for PostgreSQL instance with a private IP. Enable automated backups and high availability. Note the connection details for `DATABASE_URL`.

### 2. Redis — Memorystore [#2-redis--memorystore]

Create a Memorystore for Redis instance in the same VPC and use its endpoint for `REDIS_URL`. Enable a read replica for production.

### 3. Compute — GKE [#3-compute--gke]

Create a GKE cluster (Autopilot or Standard) sized for your traffic. GKE provisions an HTTP(S) load balancer automatically when you create an Ingress for the gateway.

### 4. Networking [#4-networking]

Use a private VPC and connect Cloud SQL via Private Service Access and Memorystore via its private endpoint. Restrict access so only the GKE workloads can reach the database and cache, and expose only the gateway to the internet.

### 5. Secrets — Secret Manager [#5-secrets--secret-manager]

Store `AUTH_SECRET`, `GATEWAY_API_KEY_HASH_SECRET`, and your provider API keys in Secret Manager. Sync them into the cluster with the [External Secrets Operator](https://external-secrets.io/) or the Secret Manager CSI driver.

## Deploy the Helm chart [#deploy-the-helm-chart]

With the backing services in place, deploy LLM Gateway with the [Helm chart](https://docs.llmgateway.io/self-host/kubernetes), pointing it at your Cloud SQL and Memorystore endpoints:

```bash
helm install llmgateway oci://ghcr.io/theopenco/charts/llmgateway -f values.yaml
```

```yaml
config:
  DATABASE_URL: "postgres://user:password@your-cloud-sql-ip:5432/llmgateway"
  REDIS_URL: "redis://your-memorystore-ip:6379"
  AUTH_SECRET: "from-secret-manager"
  GATEWAY_API_KEY_HASH_SECRET: "from-secret-manager"
```

See the [Kubernetes guide](https://docs.llmgateway.io/self-host/kubernetes) for the full set of configurable values and how to scale the gateway.

<Callout type="info">
  **Prefer not to wire this up by hand?** The [Enterprise
  plan](https://llmgateway.io/enterprise) ships Terraform modules that stand up
  the entire Google Cloud stack — GKE, Cloud SQL, Memorystore, networking, and
  secrets — and deploy LLM Gateway in one command. [Talk to
  us](https://llmgateway.io/enterprise#contact).
</Callout>


# Self Host LLM Gateway
URL: https://docs.llmgateway.io/self-host


LLM Gateway is a self-hostable platform that provides a unified API gateway for multiple LLM providers. Run it on your own infrastructure to keep full control over your data and avoid platform fees.

Pick the deployment path that matches where you're running it.

Self-hosting guides are documented under https://docs.llmgateway.io/self-host and included in full in this file.

## Which option should I choose? [#which-option-should-i-choose]

* **Trying it out or running a single low-traffic instance?** Start with [Docker](https://docs.llmgateway.io/self-host/docker) or [Docker Compose](https://docs.llmgateway.io/self-host/docker-compose) on one machine.
* **Running in production?** Deploy to [Kubernetes](https://docs.llmgateway.io/self-host/kubernetes) with our Helm chart, and use a managed Postgres and Redis from your cloud.
* **On a specific cloud?** Follow the [AWS](https://docs.llmgateway.io/self-host/aws), [Google Cloud](https://docs.llmgateway.io/self-host/gcp), or [Azure](https://docs.llmgateway.io/self-host/azure) guide for the exact managed services to provision.
* **Unlocking Enterprise features?** Install the signed token using the [Enterprise licensing guide](https://docs.llmgateway.io/self-host/enterprise-license).

## What you'll need [#what-youll-need]

Every deployment is built from the same pieces:

* **Stateless services** — the gateway, API, UI, and a background worker. Scale these freely; they hold no data between requests.
* **PostgreSQL** — the source of truth for users, projects, keys, and usage records.
* **Redis** — response caching and the queue that feeds the worker.
* **Provider API keys** — the OpenAI, Anthropic, Google, and other credentials the gateway uses, injected as secrets.

In production, run PostgreSQL and Redis as managed services from your cloud provider so backups, failover, and patching are handled for you.


# Kubernetes
URL: https://docs.llmgateway.io/self-host/kubernetes


Kubernetes is the deployment model we recommend for production. The gateway is stateless, so it scales horizontally with no coordination; pods self-heal; and the same manifests run on any cluster — EKS, GKE, AKS, or your own.

We publish an official **Helm chart** that deploys the gateway, API, UI, and worker with sane defaults and lets you wire in a managed Postgres and Redis through values.

## Prerequisites [#prerequisites]

* A Kubernetes cluster and `kubectl` configured to reach it
* [Helm 3](https://helm.sh/docs/intro/install/)
* A PostgreSQL database and Redis instance (use a managed service in production)

## Install the chart [#install-the-chart]

The chart is published as an OCI artifact on GitHub Container Registry:

```bash
helm install llmgateway oci://ghcr.io/theopenco/charts/llmgateway
```

This installs the latest published version. To pin to a specific release, append `--version <version>`, matching a published release tag without the `v` prefix (e.g. `1.2.3`).

## Configure with values [#configure-with-values]

Provide a `values.yaml` to point the chart at your managed database and cache and to set your secrets:

```yaml
config:
  AUTH_SECRET: "your-secret-key-here"
  GATEWAY_API_KEY_HASH_SECRET: "your-api-key-hash-secret-here"
  DATABASE_URL: "postgres://user:password@your-managed-host:5432/llmgateway"
  REDIS_URL: "redis://your-managed-host:6379"
  LLM_OPENAI_API_KEY: "sk-..."
  LLM_ANTHROPIC_API_KEY: "sk-ant-..."
```

```bash
helm install llmgateway oci://ghcr.io/theopenco/charts/llmgateway -f values.yaml
```

## Enterprise license [#enterprise-license]

For the chart-managed Secret, set the signed license in your values:

```yaml
enterprise:
  license: "your-signed-license"
```

For production, prefer `existingSecret` and add the key
`LLMGATEWAY_ENTERPRISE_LICENSE` to that Secret. API and gateway replicas verify
the license offline using the public key in their images. A standard license is
bound to one organization; a white-label license permits multiple Enterprise
organizations and unlocks the multi-organization admin dashboard. Rotating the
Secret requires restarting those pods. An expired license remains active for
seven days, then Enterprise features lock without affecting core gateway
availability.

See [Enterprise licensing](https://docs.llmgateway.io/self-host/enterprise-license) for license types,
Secret installation, verification, expiry behavior, and key rotation.

Provider keys support comma-separated values for load balancing and optional `__ENTERPRISE` / `__PLANS` suffix overrides used only for enterprise-plan or plan-based (DevPass/Chat plan) organizations — see [Enterprise and plan provider env overrides](https://docs.llmgateway.io/self-host/docker-compose#enterprise-and-plan-provider-env-overrides).

In production, store secrets in your cluster's secret manager rather than committing them to `values.yaml`. The cloud guides cover the recommended approach for each provider:

* [AWS](https://docs.llmgateway.io/self-host/aws) — EKS with RDS, ElastiCache, and Secrets Manager
* [Google Cloud](https://docs.llmgateway.io/self-host/gcp) — GKE with Cloud SQL, Memorystore, and Secret Manager
* [Azure](https://docs.llmgateway.io/self-host/azure) — AKS with Azure Database for PostgreSQL, Azure Cache for Redis, and Key Vault

## Scaling the gateway [#scaling-the-gateway]

Because the gateway is stateless, scale it horizontally to match traffic — either by setting the replica count in values or with a `HorizontalPodAutoscaler` that targets CPU or request load. The API, UI, and worker serve your team rather than your traffic and rarely need more than one or two replicas.

See the [Helm chart README](https://github.com/theopenco/llmgateway/tree/main/infra/helm) for the full list of configurable values and the [list of available versions](https://github.com/theopenco/llmgateway/pkgs/container/charts%2Fllmgateway).


# Gateway Caching
URL: https://docs.llmgateway.io/features/caching/gateway-caching


Gateway caching serves a previously-seen, byte-identical request entirely from LLM Gateway without forwarding it to the upstream provider. Repeated identical calls cost **$0** — there is no inference and no provider charge. It is most useful for API workloads with deterministic inputs (classification, batch jobs, FAQ lookups, retries) rather than free-form chat.

<Callout type="info">
  If you want to reduce the cost of long, partially-shared prompts in chat apps
  or coding tools, you want [Provider Cache
  Control](https://docs.llmgateway.io/features/caching/provider-cache-control) instead. That discounts the
  cached portion of your prompt on every call — it does not require
  byte-identical requests. See the [Caching Overview](https://docs.llmgateway.io/features/caching) for a
  side-by-side comparison.
</Callout>

## How It Works [#how-it-works]

When you make an API request:

1. LLM Gateway generates a cache key based on the request parameters
2. If a matching cached response exists, it's returned immediately
3. If no cache exists, the request is forwarded to the provider
4. The response is cached for future identical requests

This means repeated identical requests are served instantly from cache without incurring additional provider costs.

## Cost Savings [#cost-savings]

Caching can dramatically reduce costs for applications with repetitive requests:

| Scenario                    | Without Caching | With Caching | Savings |
| --------------------------- | --------------- | ------------ | ------- |
| 1,000 identical requests    | $10.00          | $0.01        | 99.9%   |
| 50% duplicate rate          | $10.00          | $5.00        | 50%     |
| Retry after transient error | $0.02           | $0.01        | 50%     |

<Callout type="info">
  Cached responses are free from provider costs. You only pay for the initial
  request that populates the cache.
</Callout>

## Requirements [#requirements]

<Callout type="info">
  Caching is **free** and **independent** of [Data
  Retention](https://docs.llmgateway.io/features/data-retention). Cached responses live in a short-lived
  cache (TTL-bound, typically seconds to minutes) and are not stored as
  long-term request data — you do not need to enable data retention to use
  caching.
</Callout>

To use caching:

1. Enable **Caching** in your project settings under Preferences
2. Configure the cache duration (TTL) as needed
3. Make requests as normal—caching is automatic

## Cache Key Generation [#cache-key-generation]

The cache key is generated from these request parameters:

* Model identifier
* Messages array (roles and content)
* Temperature
* Max tokens
* Top P
* Tools/functions
* Tool choice
* Response format
* System prompt
* Other model-specific parameters

<Callout type="info">
  Requests with different parameter values, even slight variations, will not
  share cache entries.
</Callout>

## Cache Behavior [#cache-behavior]

### Cache Hits [#cache-hits]

When a cache hit occurs:

* Response is returned immediately (sub-millisecond latency)
* No provider API call is made
* No inference costs are incurred

### Cache Misses [#cache-misses]

When a cache miss occurs:

* Request is forwarded to the LLM provider
* Response is stored in cache
* Normal inference costs apply
* Future identical requests will hit the cache

## Streaming and Caching [#streaming-and-caching]

Caching works with both streaming and non-streaming requests:

* **Non-streaming**: Full response is cached and returned
* **Streaming**: The complete response is reconstructed from cache and streamed back

## Cache TTL (Time-to-Live) [#cache-ttl-time-to-live]

Cache duration is configurable per project in your project settings. You can set the cache TTL from 10 seconds up to 1 year (31,536,000 seconds).

The default cache duration is 60 seconds. Adjust this based on your use case—longer durations work well for static content, while shorter durations are better for frequently changing data.

## Identifying Cached Responses [#identifying-cached-responses]

A cache hit replays the stored response body, so it is byte-identical to the original — same `id`, same content. Two markers tell you it was a replay:

* the `x-llmgateway-cache: HIT` response header (set on every lane, including `/v1/messages`, which has no metadata envelope)
* `metadata.cached: true` on the response body (and on the final metadata chunk of a streamed replay)

All cost fields are zeroed on a replay, because no upstream call was made:

```json
{
	"usage": {
		"prompt_tokens": 12,
		"completion_tokens": 48,
		"total_tokens": 60,
		"cost": 0,
		"cost_details": {
			"total_cost": 0,
			"input_cost": 0,
			"output_cost": 0
		}
	},
	"metadata": {
		"cached": true
	}
}
```

Token counts are **not** zeroed — they still describe the completion you are receiving, and are what the dashboard records for analytics. Note that any prompt-caching fields (`prompt_tokens_details.cached_tokens`, `cache_write_tokens`) describe the *original* upstream call, not the replay.

## Bypassing the Cache for a Single Request [#bypassing-the-cache-for-a-single-request]

Send `x-no-cache: true` to skip the cache for one request — the call goes upstream and its response is not stored. Useful when a client retries an identical request and expects a fresh sample (for example an agent loop) without disabling caching for the whole project.

```bash
curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-no-cache: true" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello"}]}'
```

## Use Cases [#use-cases]

### Development and Testing [#development-and-testing]

During development, you often send the same prompts repeatedly:

```typescript
// This prompt will only incur costs once
const response = await client.chat.completions.create({
	model: "gpt-4o",
	messages: [{ role: "user", content: "Explain quantum computing" }],
});
```

### Chatbots with Common Questions [#chatbots-with-common-questions]

FAQ-style interactions often have repeated questions:

```typescript
// Common questions are served from cache
const faqs = [
	"What are your business hours?",
	"How do I reset my password?",
	"What is your return policy?",
];
```

### Batch Processing [#batch-processing]

Processing large datasets with potentially duplicate items:

```typescript
// Duplicate items in batch are served from cache
for (const item of items) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages: [{ role: "user", content: `Classify: ${item}` }],
	});
}
```

## Best Practices [#best-practices]

### Maximize Cache Hits [#maximize-cache-hits]

* Use consistent prompt formatting
* Normalize input data before sending
* Use deterministic parameters (temperature: 0)
* Avoid including timestamps or random values in prompts

### Appropriate Use Cases [#appropriate-use-cases]

Caching is most effective for:

* Static knowledge queries
* Classification tasks
* FAQ responses
* Development/testing
* Retry scenarios

### When to Avoid Caching [#when-to-avoid-caching]

Caching may not be suitable for:

* Real-time data requirements
* Highly personalized responses
* Time-sensitive information
* Creative tasks requiring variety
* Chat or coding tools where prompts overlap but are not byte-identical — use [Provider Cache Control](https://docs.llmgateway.io/features/caching/provider-cache-control) instead

## Pricing [#pricing]

Caching is **completely free**. Cached responses are held in a short-lived
in-memory cache (bounded by your configured TTL) and do not incur storage
charges. Storage costs only apply if you separately enable [Data
Retention](https://docs.llmgateway.io/features/data-retention) for full request/response payloads.

<Callout type="success">
  Caching reduces both inference cost and latency at no additional charge.
</Callout>


# Caching
URL: https://docs.llmgateway.io/features/caching


LLM Gateway supports **two distinct kinds of caching**, and they solve different problems. Pick the one that matches your workload — they can also be used together.

## Provider / Model Caching [#provider--model-caching]

The provider performs the caching. When your request reuses a long prefix from a previous call (a system prompt, conversation history, tool definitions, a long document), the model serves that prefix from its prompt cache and bills it at a reduced rate. New input tokens and **all output tokens are still billed at the normal rate** — only the cached portion is discounted.

This is the type of caching that powers efficient chat-based and assistant-based interactions, including chat apps and coding tools (Cursor, Cline, Claude Code, etc.) where the same context is reused turn after turn.

You see it in your usage as `prompt_tokens_details.cached_tokens`. For most providers it works automatically; some (notably Anthropic) also let you mark blocks explicitly with `cache_control` and choose a longer TTL.

→ &#x2A;*[Read the Provider Cache Control docs](https://docs.llmgateway.io/features/caching/provider-cache-control)**

## Gateway Caching [#gateway-caching]

LLM Gateway performs the caching. When a request is **byte-identical*&#x2A; to a previous one (same model, same messages, same parameters), the response is served from the gateway's cache without any provider call. Repeated identical calls cost **$0**.

This is most useful for deterministic API workloads — classification, batch jobs, FAQ lookups, retries — rather than free-form chat, because chat prompts almost always differ on the latest turn.

→ &#x2A;*[Read the Gateway Caching docs](https://docs.llmgateway.io/features/caching/gateway-caching)**

## Which one do I want? [#which-one-do-i-want]

| If you…                                                         | Use                                                                                           |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Build a chat app, assistant, or coding tool                     | [Provider Cache Control](https://docs.llmgateway.io/features/caching/provider-cache-control)                            |
| Send long system prompts or growing conversation history        | [Provider Cache Control](https://docs.llmgateway.io/features/caching/provider-cache-control)                            |
| Want longer cache lifetimes than the provider default           | [Provider Cache Control](https://docs.llmgateway.io/features/caching/provider-cache-control) (explicit `cache_control`) |
| Send the exact same request many times (batches, retries, FAQs) | [Gateway Caching](https://docs.llmgateway.io/features/caching/gateway-caching)                                          |
| Want $0 on repeated calls instead of a discount                 | [Gateway Caching](https://docs.llmgateway.io/features/caching/gateway-caching)                                          |

<Callout type="info">
  The two are not mutually exclusive. A coding tool can rely on provider caching
  for its long system prompt **and** enable gateway caching so that
  deterministic tool calls (e.g., file lookups) cost nothing on retry.
</Callout>


# Provider Cache Control
URL: https://docs.llmgateway.io/features/caching/provider-cache-control


Most modern LLM providers offer **prompt caching**: when a request reuses a long prefix from a previous request (for example, a multi-thousand-token system prompt or a growing conversation history), the provider stores that prefix and serves it back at a steep discount on subsequent calls. Only the cached portion is discounted — new input tokens and all output tokens are still billed at the normal rate.

This is the behavior you see surfaced as `cached_tokens` in your usage payloads, and it is what makes chat apps, assistants, and coding tools (Cursor, Cline, Claude Code, etc.) economically viable on long contexts.

<Callout type="info">
  Looking for $0 on repeated calls instead of a discount on the cached portion?
  That is [Gateway Caching](https://docs.llmgateway.io/features/caching/gateway-caching), which serves
  byte-identical requests entirely from LLM Gateway without hitting the
  provider. It is a better fit for deterministic API workloads than for chat.
  See the [Caching Overview](https://docs.llmgateway.io/features/caching) for a side-by-side comparison.
</Callout>

## Automatic caching [#automatic-caching]

For most users, prompt caching just works — you do not need to change your request payloads.

Providers including OpenAI, Anthropic (when prompts cross the provider's minimum size), Google, DeepSeek, xAI, and Alibaba inspect incoming requests for shared prefixes and cache them automatically. LLM Gateway forwards the provider's cache metadata back to you in the response, and bills the cached portion at the model's `cached_input` rate.

For **Anthropic** and **AWS Bedrock Claude**, prompt caching is strictly opt-in via `cache_control` / `cachePoint` markers on the request body. To get automatic cache benefits without rewriting your requests, LLM Gateway injects those markers for you on long system and user messages by default.

### Choosing a cache-write mode [#choosing-a-cache-write-mode]

**Project Settings → Caching → Provider Cache Writes** (on DevPass, **Settings → Provider cache writes**) controls how the gateway treats those markers:

| Mode                    | Markers your client sends | Markers the gateway adds |
| ----------------------- | ------------------------- | ------------------------ |
| **Automatic** (default) | Forwarded                 | Added on long prompts    |
| **Client-managed**      | Forwarded                 | Never added              |
| **Disabled**            | Stripped                  | Never added              |

**Automatic** suits requests that do not manage caching themselves. If you send long prompts sporadically — with gaps wider than the 5-minute TTL — you pay the cache-write premium (1.25× input for 5m, 2× for 1h) without ever benefiting from a cache read, so one of the other modes will be cheaper.

**Client-managed** hands the decision to each request: a request writes to the provider cache only if it carries its own markers. Use it when one API key serves both a coding tool that sets its own markers (Claude Code, Cursor, Cline) and other traffic that should not pay the write premium — Automatic would add markers to the latter, and Disabled would remove the former's.

**Disabled** turns provider caching off for the project entirely, including markers your client sends.

The mode applies to every upstream that takes an explicit cache marker, whichever field it uses for one — the gateway translates your `cache_control` into the marker the resolved provider expects. Providers that cache automatically with no marker at all are unaffected, since there is nothing to forward or strip. On models whose explicit caching is a request-level mode rather than a per-block marker, Client-managed and Disabled also send `prompt_cache_options: {"mode": "explicit"}`, so implicit caching does not write a cache the request never asked for. The [models page](https://llmgateway.io/models) shows which models support prompt caching.

Where the upstream takes per-block markers, they are forwarded on `system` blocks, message text blocks, tool definitions, and `tool_result` blocks. A `cache_control` on an image or other non-text content block is not forwarded — put the breakpoint on an adjacent text block instead.

Changes take up to 5 minutes to take effect due to the project-settings cache.

To take advantage of automatic caching:

* Put stable content (system prompt, instructions, tool definitions, long documents) at the **start** of your messages
* Keep the variable portion (the latest user turn) at the **end**
* Reuse the same prefix across requests — even minor changes invalidate the cache

You can confirm the cache is working by inspecting `usage.prompt_tokens_details.cached_tokens` on the response. See [Cost Breakdown](https://docs.llmgateway.io/features/cost-breakdown) for the full list of usage fields.

```json
{
	"usage": {
		"prompt_tokens": 8200,
		"completion_tokens": 150,
		"prompt_tokens_details": {
			"cached_tokens": 8000
		},
		"cost_details": {
			"input_cost": 0.0006,
			"cached_input_cost": 0.0008
		}
	}
}
```

In this example, 8,000 of the 8,200 prompt tokens were served from the provider's cache and billed at the cached rate.

### Pricing and routing [#pricing-and-routing]

Cached input tokens are billed at the model's published `cached_input` price (typically 10–25% of the regular input price, depending on the provider and model). Output tokens and any non-cached input tokens are billed at the normal rate.

When the [Smart Routing](https://docs.llmgateway.io/features/routing) algorithm selects a provider for a large prompt (≥ 5,000 estimated tokens), it gives extra weight to providers that advertise cache support, since caching can substantially reduce the cost of repeated large prompts.

## Explicit caching with `cache_control` [#explicit-caching-with-cache_control]

Some providers — most notably **Anthropic** — also support *explicit* cache control, where you mark specific content blocks as cacheable using a `cache_control` field. This gives you precise control over what gets cached and lets you opt into longer cache lifetimes than the default.

Explicit caching is provider-specific. Supported providers and TTLs at the time of writing:

| Provider             | Models                         | Supported TTLs       |
| -------------------- | ------------------------------ | -------------------- |
| Anthropic (Claude)   | All Claude models              | `5m` (default), `1h` |
| AWS Bedrock (Claude) | All Claude models              | `5m` (default), `1h` |
| Alibaba (Qwen)       | Qwen models with cache support | Provider-defined     |

To mark content as cacheable, send the message content as an array of blocks and add a `cache_control` field to the block you want to cache:

```json
{
	"model": "claude-haiku-4-5",
	"messages": [
		{
			"role": "system",
			"content": [
				{
					"type": "text",
					"text": "You are a helpful assistant. <long instructions...>",
					"cache_control": { "type": "ephemeral", "ttl": "1h" }
				}
			]
		},
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
}
```

Use `ttl: "5m"` (the default if omitted) for short-lived caches that match a single user's session, and `ttl: "1h"` when the same prefix will be reused over a longer window (for example, a coding agent that keeps the same project context warm across many requests).

The same `cache_control` markers work on the native [`/v1/messages` endpoint](https://docs.llmgateway.io/features/anthropic-endpoint) (Anthropic request format), where cache usage comes back in Anthropic's own fields: `usage.cache_creation_input_tokens` (written this request) and `usage.cache_read_input_tokens` (served from cache).

### Minimum cacheable prompt length [#minimum-cacheable-prompt-length]

Every Claude model has a **minimum prompt length below which nothing is cached**. A `cache_control` marker on a shorter prompt is accepted without error, but the provider silently skips the cache write — the response reports `cache_creation_input_tokens: 0` and `cache_read_input_tokens: 0` on every call, no matter how often you repeat the request. This is Anthropic's documented behavior, not a dropped breakpoint; you would see exactly the same result calling Anthropic directly.

The threshold counts **all tokens up to and including the marked block** (tools, system, and preceding messages), and it varies by model generation:

| Models                                                | Minimum cacheable tokens |
| ----------------------------------------------------- | ------------------------ |
| Opus 4.5+, Sonnet 5, Haiku 4.5                        | 4,096                    |
| Sonnet 4.6, Haiku 3.5                                 | 2,048                    |
| Sonnet 4.5 and earlier, Opus 4.1 and earlier, Haiku 3 | 1,024                    |

The models endpoint (`GET https://api.llmgateway.io/v1/models`) exposes the exact threshold as `min_cacheable_tokens` on each provider entry, so clients can check it programmatically. LLM Gateway's automatic marker injection uses the same threshold, which is why short system prompts never trigger automatic cache writes either.

### Verifying cache writes and the write premium [#verifying-cache-writes-and-the-write-premium]

The first request that creates a cache entry is billed at the provider's cache-write rate (1.25× input for `5m`, 2× for `1h`). On the OpenAI-compatible `/v1/chat/completions` endpoint, LLM Gateway surfaces the write side in extended usage fields — OpenAI's standard format only has `cached_tokens` for reads, so these are gateway extensions:

```json
{
	"usage": {
		"prompt_tokens": 5232,
		"prompt_tokens_details": {
			"cached_tokens": 0,
			"cache_write_tokens": 5222,
			"cache_creation_tokens": 5222,
			"cache_creation": {
				"ephemeral_5m_input_tokens": 5222,
				"ephemeral_1h_input_tokens": 0
			}
		},
		"cost_details": {
			"input_cost": 0.00003,
			"cache_write_input_cost": 0.0065275,
			"cached_input_cost": 0
		}
	}
}
```

`cache_write_tokens` (and its alias `cache_creation_tokens`) counts the prompt tokens written into the provider cache this request, `cache_creation` breaks the write down by TTL when both rates are in play, and `cost_details.cache_write_input_cost` is the exact USD amount billed at the write premium. A successful cache write on call 1 shows up as `cache_write_tokens > 0`; the matching read on call 2 shows up as `cached_tokens > 0` with `cached_input_cost` at the discounted rate.

### Mixing explicit markers with automatic injection [#mixing-explicit-markers-with-automatic-injection]

Anthropic requires cache breakpoints with longer TTLs to appear before shorter ones (blocks are processed in the order `tools`, `system`, `messages`). The markers LLM Gateway injects automatically use the default 5-minute TTL, so they could never legally precede an explicit `ttl: "1h"` marker in your messages. To keep both features compatible:

* When your request contains an explicit `ttl: "1h"` marker in the **messages**, LLM Gateway skips its automatic marker injection for that request entirely and forwards only your markers — the same behavior you would get calling the provider directly.
* A `ttl: "1h"` marker only on the **system** prompt does not disable automatic injection, since 5-minute breakpoints after it still satisfy the ordering rule.
* Explicit markers that use the default 5-minute TTL coexist with automatic injection (capped at 4 breakpoints total per Anthropic's limit).

This section describes the default **Automatic** mode. In **Client-managed** mode there is no injection to reconcile — your markers are forwarded exactly as sent, whatever their TTL.

<Callout type="warning">
  Cache writes are billed at a premium (typically 1.25x for 5m and 2x for 1h on
  Anthropic) the first time a cached block is created. After that, cache reads
  cost roughly 10% of the regular input price. The break-even point is usually
  one or two reuses — explicit caching is worth it whenever a marked block will
  be sent more than once within its TTL.
</Callout>

Anthropic returns a per-TTL breakdown of cache writes when you mix `5m` and `1h` blocks:

```json
{
	"usage": {
		"cache_creation": {
			"ephemeral_5m_input_tokens": 0,
			"ephemeral_1h_input_tokens": 8000
		},
		"cache_read_input_tokens": 0
	}
}
```

For providers that publish a separate explicit-cache read rate (for example, Alibaba Qwen charges 10% for explicit cache reads vs. 20% for automatic cache reads), LLM Gateway detects the `cache_control` markers on your request and applies the explicit rate automatically.

## Related [#related]

* [Gateway Caching](https://docs.llmgateway.io/features/caching/gateway-caching) — serve byte-identical requests entirely from LLM Gateway at $0 cost
* [Caching Overview](https://docs.llmgateway.io/features/caching) — side-by-side comparison of provider caching vs. gateway caching
* [Cost Breakdown](https://docs.llmgateway.io/features/cost-breakdown) — full reference for the usage and cost fields on every response
* [Smart Routing](https://docs.llmgateway.io/features/routing) — how cache support influences provider selection for large prompts


# Microsoft Entra ID
URL: https://docs.llmgateway.io/features/sso/entra


This guide walks an organization admin through connecting **Microsoft Entra ID** (formerly Azure Active Directory) to LLM Gateway for SAML single sign-on and SCIM provisioning.

<Callout type="info">
  SSO & SCIM require the **Enterprise** plan and an organization **owner** or
  **admin**. See the [SSO overview](https://docs.llmgateway.io/features/sso) for prerequisites and how the
  URLs are derived.
</Callout>

## Finding "Enterprise applications" in Entra [#finding-enterprise-applications-in-entra]

An **Enterprise application** is a registered instance of an app that your tenant trusts for sign-in. You'll create one to represent LLM Gateway; its &#x2A;*Single sign-on (SAML)** page is where you exchange URLs and download the certificate.

<Callout type="warn">
  This is **not** the same as **App registrations** (that section is for apps
  you are developing). Use **Enterprise applications**. They are siblings under
  **Applications** and easy to confuse.
</Callout>

<Steps>
  <Step>
    ### Open the Entra admin center [#open-the-entra-admin-center]

    Go to [entra.microsoft.com](https://entra.microsoft.com) and sign in with an account that has at least the **Cloud Application Administrator** or **Application Administrator** role (Global Administrator also works). A regular member account won't see these options.
  </Step>

  <Step>
    ### Navigate to Enterprise applications [#navigate-to-enterprise-applications]

    In the left-hand navigation, click **Entra ID** (older tenants label this group **Identity**), then expand **Applications** and click **Enterprise applications** (older label: **Enterprise apps**).

    You're now on the **All applications** list.

    <Callout type="info">
      Prefer the classic Azure portal? Go to
      [portal.azure.com](https://portal.azure.com) → search **Microsoft Entra ID** →
      open it → left menu **Enterprise applications**. Both portals reach the same
      app.
    </Callout>
  </Step>

  <Step>
    ### Create the application [#create-the-application]

    Click **+ New application*&#x2A; → **+ Create your own application**. Name it (e.g. `LLM Gateway&#x60;), choose &#x2A;*"Integrate any other application you don't find in the gallery (Non-gallery)"**, then **Create**. Wait a few seconds for it to provision.

    <Callout type="warn">
      Don't search the Entra **gallery** for "LLM Gateway" — there is no gallery
      listing, so always use **Create your own application → Non-gallery**. You
      configure SAML and SCIM manually with the URLs from your **SSO** page; the
      steps below cover everything.
    </Callout>
  </Step>

  <Step>
    ### Open the SAML SSO page [#open-the-saml-sso-page]

    On the app's **Overview**, use the app's own left menu → **Single sign-on**. On the **Select a single sign-on method** screen, choose **SAML**.

    This opens the 5-section **Set up Single Sign-On with SAML** page referenced throughout the rest of this guide.
  </Step>
</Steps>

## Configure SAML SSO [#configure-saml-sso]

Because Entra needs LLM Gateway's URLs and LLM Gateway needs Entra's Login URL and certificate, the two systems exchange values. Follow this order.

<Steps>
  <Step>
    ### Choose a connection slug [#choose-a-connection-slug]

    In LLM Gateway (**Organization → SSO → Add a connection**), set **Identity provider** to **Microsoft Entra ID**, then pick a **connection slug** (lowercase letters, numbers, hyphens; `^[a-z0-9-]+$`, up to 64 chars). We recommend the format `<your-org-slug>-<provider>`, e.g. `acme-entra` — the form pre-fills this from your organization name and the selected provider, so you can accept it or edit it. It must be unique across all LLM Gateway organizations, and each organization has exactly one SSO connection. Keep it stable: your SP URLs are built from it.

    As soon as the slug is set, the form shows your two SP URLs (with the hosted default `https://internal.llmgateway.io`) — copy them from there:

    * **Identifier (Entity ID):** `https://internal.llmgateway.io/auth/sso/saml2/sp/metadata?providerId=acme-entra`
    * **Reply URL (ACS):** `https://internal.llmgateway.io/auth/sso/saml2/sp/acs/acme-entra`

    You'll paste these into Entra next — no need to create the connection yet.
  </Step>

  <Step>
    ### Set Basic SAML Configuration in Entra [#set-basic-saml-configuration-in-entra]

    On the Entra &#x2A;*Single sign-on (SAML)** page, **Section 1 – Basic SAML Configuration** → **Edit**:

    * Under &#x2A;*Identifier (Entity ID)**, click **Add identifier** and paste your metadata URL from the previous step.
    * Under &#x2A;*Reply URL (Assertion Consumer Service URL)**, click **Add reply URL** and paste your ACS URL.
    * Leave **Sign on URL**, **Relay State**, and **Logout Url** blank.

    Click **Save** (the button enables once both required fields are filled), then close the panel.

    <Callout type="warn">
      The Reply URL row has a small numeric **Index** column next to the URL field.
      Paste the ACS URL into the **Enter a reply URL*&#x2A; box only — if it lands in the
      Index box you'll see &#x2A;"The value must be a valid number."* Leave Index blank.
    </Callout>
  </Step>

  <Step>
    ### Confirm the claims [#confirm-the-claims]

    **Section 2 – Attributes & Claims**: the Entra defaults are fine. In **Microsoft Entra ID** mode, LLM Gateway reads email/name from the standard claim URIs Entra sends by default (`.../emailaddress`, `.../givenname`, `.../surname`, `.../name`), so you don't need to force the Name ID to be the email — just make sure users have a real `mail`/UPN.
  </Step>

  <Step>
    ### Grab the certificate and Login URL [#grab-the-certificate-and-login-url]

    * **Section 3 – SAML Certificates** → next to &#x2A;*Certificate (Base64)**, click **Download**. The `.cer` file's contents are a PEM block (`-----BEGIN CERTIFICATE----- … -----END CERTIFICATE-----`). Use &#x2A;*Certificate (Base64)**, not Raw or the Federation Metadata XML.
    * **Section 4 – Set up \<app name>** → copy the **Login URL** (looks like `https://login.microsoftonline.com/<tenant-id>/saml2`).

    Keep both values handy for the next step.

    <Callout type="info">
      The certificate **Download** links stay greyed out until Section 1's required
      fields are filled and saved — so complete the previous step first. The
      auto-generated signing certificate is valid for three years; its thumbprint
      and expiry are shown in Section 3.
    </Callout>
  </Step>

  <Step>
    ### Create the connection in LLM Gateway [#create-the-connection-in-llm-gateway]

    Back on the **SSO → Add a connection** form:

    * **Identity provider:** **Microsoft Entra ID** — important, this switches on the Entra claim mapping and relaxes the NameID format
    * **Connection slug:** `acme-entra` (must match the slug whose URLs you pasted into Entra)
    * **Email domains:** your corporate domain(s), e.g. `acme.com` (comma-separate for several)
    * **Identity Provider Single Sign-On URL:** the Entra **Login URL**
    * **X.509 signing certificate:** paste the PEM block

    Save. The connection card then shows the exact SP metadata + ACS URLs — confirm they match what you entered in Entra **Section 1**. You can edit the email domain list on the connection card later without re-creating the connection.

    <Callout type="warn">
      The domain list must include **every domain Entra may send as a user's
      email**. LLM Gateway signs a user in from the `emailaddress&#x60; claim, which
      Entra fills from the &#x2A;*`mail`** attribute — and that domain can differ from
      the sign-in (userPrincipalName) domain, e.g. UPN `jane@acme.local` with mail
      `jane@acme.com`. If they differ, list **both** (`acme.local, acme.com`);
      otherwise sign-in bounces back to the login page with an `account_not_linked`
      error even though Entra authenticated the user.
    </Callout>
  </Step>
</Steps>

## Assign users to the app [#assign-users-to-the-app]

Configuring SAML isn't enough on its own — Entra only issues a SAML assertion for users who are **assigned** to the application. On the Enterprise App → **Users and groups → + Add user/group**, add the people (or groups) who should be able to sign in, then **Assign**.

<Callout type="warn">
  If you skip this, sign-in fails at Entra with &#x2A;"…is not assigned to a role for
  the application"* (error `AADSTS50105`) — even though the SAML config is
  correct. Assign at least your own account before testing.
</Callout>

If you also set up SCIM below, these same assignments determine which users and groups are in provisioning scope. For SSO **without** SCIM this assignment step is still mandatory.

## Configure SCIM provisioning [#configure-scim-provisioning]

SSO only signs people in. SCIM is what actually adds members to the organization. Set it up on the **same** Enterprise App.

<Steps>
  <Step>
    ### Generate a SCIM token [#generate-a-scim-token]

    In LLM Gateway, on &#x2A;*Organization → SSO → Directory sync (SCIM)**, click **Generate SCIM token** and copy it (shown once). The SCIM base URL is shown in the same card — it's `https://internal.llmgateway.io/scim/v2` on the hosted default.
  </Step>

  <Step>
    ### Add admin credentials in Entra [#add-admin-credentials-in-entra]

    In the current Entra provisioning workspace, open your Enterprise App → **Provisioning → Manage → Connectivity**. In the classic interface, use **Provisioning → Automatic → Admin Credentials** instead:

    * **Authentication method:*&#x2A; choose the token option — labelled &#x2A;*"Long-lived bearer token"*&#x2A; (or &#x2A;*"Secret token"**). **Do not** pick "OAuth 2.0 code grant flow": LLM Gateway's SCIM endpoint authenticates a plain static bearer token (`Authorization: Bearer <token>`), not an OAuth code exchange. The presence of a **Tenant URL + Secret token** pair on the form confirms you're on the right method.
    * **Tenant URL:** the SCIM base URL, e.g. `https://internal.llmgateway.io/scim/v2`
    * **Secret Token:** the token from the previous step
    * Click **Test Connection** (Entra calls `/ServiceProviderConfig` with the bearer token), then **Save**.

    A green check confirms the credentials are valid.

    <Callout type="info">
      Entra rolled out a redesigned provisioning UX that adds the &#x2A;*"Select
      authentication method"** dropdown shown above. The classic UX has no dropdown
      and just shows Tenant URL + Secret token (which is the bearer method). Either
      is fine — pick the bearer/secret-token option and the steps are identical.
    </Callout>

    <Callout type="warn">
      Do not continue until **Test connection** succeeds. The Tenant URL must be
      publicly reachable by Microsoft; a stopped local server or expired temporary
      tunnel leaves Entra configured but unable to sync.
    </Callout>
  </Step>

  <Step>
    ### Assign users and start provisioning [#assign-users-and-start-provisioning]

    * Under **Manage → Provisioning → Mappings**, confirm both **Provision Microsoft Entra ID Users** and **Provision Microsoft Entra ID Groups** show **Yes**.
    * Under **Manage → Attribute mapping**, the default user mappings are fine (`userPrincipalName → userName`, `mail → emails[work]`, `givenName/surname → name.*`, `accountEnabled → active`).
    * Expand **Settings** and set **Scope** to **Sync only assigned users and groups**.
    * Under **Users and groups**, assign the people/groups who should get access.
    * Set **Provisioning Status = On** and Save. Use **Provision on demand** to test a single user instantly.

    Entra then begins syncing assigned users on its provisioning cycle.

    <Callout type="info">
      If the Overview says **Initial sync paused** and **Last cycle completed:
      Never**, return to **Manage → Provisioning**, set **Provisioning Status** to
      **On**, and click **Save**.
    </Callout>
  </Step>
</Steps>

Assigned users are created in LLM Gateway and added to the organization (default role **developer**). Deactivating or unassigning a user in Entra removes their membership.

## Sync Entra groups to LLM Gateway teams [#sync-entra-groups-to-llm-gateway-teams]

The complete path is:

`Entra security group → Enterprise App assignment → SCIM group + members → exact-name mapping → LLM Gateway team policy`

Three Entra settings are involved, and they do different jobs:

| Entra setting                                                         | Purpose                                                                                                                  | If it is missing                                                                   |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------- |
| **Security group membership**                                         | Defines who belongs to the group.                                                                                        | Entra has no membership change to send.                                            |
| **Enterprise App → Users and groups** assignment                      | Places the group and its immediate members in scope when provisioning is set to **Sync only assigned users and groups**. | The group is skipped as not assigned.                                              |
| **Provisioning → Manage → Provisioning → Mappings** group job enabled | Sends the group object, exact `displayName`, and membership changes to LLM Gateway.                                      | Users may be provisioned, but LLM Gateway never sees their Entra group membership. |

<Callout type="warn">
  Assigning a group to the Enterprise App is not the same as provisioning the
  group object. Team mapping works only when the group is assigned **and** the
  group attribute mapping is enabled.
</Callout>

<Callout type="info">
  You do not need to add a groups claim to the SAML assertion. SAML handles
  sign-in; SCIM sends the group object and membership used for team mapping.
</Callout>

Microsoft documents [Enterprise App group assignment](https://learn.microsoft.com/entra/identity/enterprise-apps/assign-user-or-group-access-portal) and [group object provisioning](https://learn.microsoft.com/entra/identity/app-provisioning/customize-application-attributes#editing-group-attribute-mappings) separately for the same reason.

### Recommended group layout [#recommended-group-layout]

Use one unmapped access group to keep users provisioned, plus separate groups for team policy:

| Entra security group | Assigned to the Enterprise App | LLM Gateway mapping           | Purpose                                                                     |
| -------------------- | ------------------------------ | ----------------------------- | --------------------------------------------------------------------------- |
| `Gateway Access`     | Yes                            | None                          | Baseline SSO/SCIM access; keeps the user in scope while their team changes. |
| `Gateway Platform`   | Yes                            | `Gateway Platform → Platform` | Applies the Platform team's policy.                                         |
| `Gateway Data`       | Yes                            | `Gateway Data → Data`         | Applies the Data team's policy.                                             |

Make every user an **immediate member** of `Gateway Access` and an immediate member of the appropriate policy group. Do not nest the policy groups inside the access group: Entra application assignment and provisioning do not expand nested group membership.

<Callout type="warn">
  Without the baseline group (or a direct user assignment), leaving a mapped
  policy group can make the user entirely out of scope. Entra then deprovisions
  the organization member instead of only removing their team, which also
  revokes that member's API keys.
</Callout>

<Steps>
  <Step>
    ### Create the Entra security groups [#create-the-entra-security-groups]

    In the Entra admin center, go to **Entra ID → Groups → All groups → New group**:

    * **Group type:** **Security**. Entra requires assigned groups to be security-enabled.
    * **Group name:** choose a stable name such as `Gateway Platform`. LLM Gateway matches this display name exactly, including case and spaces.
    * **Membership type:** **Assigned** is simplest. Dynamic membership also works, but its own evaluation delay is added before provisioning runs.
    * Add the users as direct members.

    Group-based Enterprise App assignment requires Microsoft Entra ID P1 or P2.
  </Step>

  <Step>
    ### Assign every relevant group to the Enterprise App [#assign-every-relevant-group-to-the-enterprise-app]

    Open **Entra ID → Enterprise apps → All applications → \<your LLM Gateway app> → Users and groups → Add user/group**. Select the baseline access group and every group you plan to map, keep the offered application role (commonly **User** or **Default Access**), then click **Assign**. That role does not select an LLM Gateway team; the exact-name mapping does.

    Assigning only individual users is enough to provision those users, but it does not put an unassigned group object in scope. The group itself must appear on the Enterprise App's **Users and groups** page.
  </Step>

  <Step>
    ### Enable group object provisioning [#enable-group-object-provisioning]

    On the same Enterprise App, open **Provisioning → Manage → Provisioning**, expand **Mappings**, and confirm **Provision Microsoft Entra ID Groups** shows **Yes**. Then open **Manage → Attribute mapping → Groups** and confirm the three default group mappings remain:

    * `displayName → displayName`
    * `members → members`
    * `objectId → externalId`

    In the classic interface, these controls are together under **Provisioning → Mappings → Provision Microsoft Entra ID Groups**. Keep **Settings → Scope** set to **Sync only assigned users and groups**, set **Provisioning Status** to **On**, and save.
  </Step>

  <Step>
    ### Map the exact group name in LLM Gateway [#map-the-exact-group-name-in-llm-gateway]

    First create the destination team under **Organization → Teams**. Then open **Organization → SSO → Group team mapping**:

    1. Enter the Entra group's exact display name, for example `Gateway Platform`.
    2. Select the LLM Gateway team, for example **Platform**.
    3. Click **Save mapping**.

    The mapping can be created before or after Entra pushes the group. If the group already exists in LLM Gateway, saving applies the team immediately to its current developer members.
  </Step>

  <Step>
    ### Run and verify the first sync [#run-and-verify-the-first-sync]

    For a quick test, use **Enterprise App → Provisioning → Provision on demand**, search for the group, select an affected member, then click **Provision**. Entra's [on-demand group test](https://learn.microsoft.com/entra/identity/app-provisioning/provision-on-demand) handles up to five selected members at a time. For the normal background cycle, use **Start provisioning** and check **Provisioning logs** for the group and its users; Entra generally runs the synchronization process about every 40 minutes.

    In LLM Gateway, open **Organization → Members** and confirm the developer shows the expected team. Provisioning only the user tests user creation; provision the **group** to test group-to-team mapping.
  </Step>
</Steps>

### What changes after the first sync [#what-changes-after-the-first-sync]

| Change in Entra                   | SCIM change sent to LLM Gateway                 | Result                                                                                             |
| --------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Add an assigned group/member      | Create or update the user and group membership. | The developer receives the mapped team.                                                            |
| Move a user between mapped groups | Remove one group membership and add the other.  | The developer moves to the newly resolved team.                                                    |
| Remove the last mapped membership | Remove the group membership.                    | The SCIM-assigned team is cleared; the user remains provisioned through the baseline access group. |
| Rename a mapped group             | Update the SCIM group's `displayName`.          | The old exact-name mapping stops matching; update it on the LLM Gateway SSO page.                  |
| Unassign or delete the group      | Deprovision the SCIM group.                     | Its members lose that mapped team; users remain only if another assignment keeps them in scope.    |

An explicit non-null team assignment made by an admin on the **Members** page is never overwritten. If an admin only clears a SCIM-assigned team, the next SCIM event restores the mapped team. If several mapped groups apply, the alphabetically first exact group name wins. Owners and admins cannot belong to teams, so a privileged group → role mapping takes precedence.

<Callout type="info">
  Team project access is an additional ceiling: a developer can use a project
  only when both their personal/default project grants and their team allow it.
</Callout>

### If the user syncs but the team does not [#if-the-user-syncs-but-the-team-does-not]

Check the path in order:

1. On the Enterprise App's **Users and groups** page, confirm the **group** is assigned—not only its individual user.
2. Under **Provisioning → Manage → Provisioning → Mappings**, confirm **Provision Microsoft Entra ID Groups** is **Yes**.
3. Under **Manage → Attribute mapping → Groups**, confirm `displayName`, `members`, and `objectId` are still mapped.
4. Use **Provision on demand** with the group and member, then inspect the **Provisioning logs** for a group operation.
5. In LLM Gateway, confirm the Group team mapping is character-for-character identical to the Entra display name and the member's role is **developer**.

If the user appears in LLM Gateway but no group operation appears in Entra's logs, the problem is on the Entra assignment or group-provisioning side. If the group operation succeeds but the team stays empty, check the exact-name mapping and member role in LLM Gateway.

## Group → role mapping (optional) [#group--role-mapping-optional]

The same SCIM groups can drive organization roles. On the **SSO** page, add mappings such as `Gateway Admins → admin`. Each member gets the **highest** mapped role among their groups (default `developer`). Owners are never auto-demoted. Because owners and admins cannot belong to teams, a privileged role mapping clears any team assignment.

## Default project access (optional) [#default-project-access-optional]

Provisioned members join as **developers**, who only see the projects they're granted (owners/admins always see every project). Use the **Default project access** card on the **SSO** page to choose which projects newly provisioned SSO/SCIM developers get by default. If you leave it unconfigured, new members get your organization's first (default) project so they can see something out of the box; select an empty set to instead grant nothing and manage project access manually per member.

<Callout type="info">
  Default project access applies only to members provisioned **after** you set
  it — it doesn't retroactively change existing members' project grants. Adjust
  an existing member's projects from the **Team** page.
</Callout>

## Test the login [#test-the-login]

Log out → login page → **Sign in with SSO** → enter a work email (`someone@acme.com`). LLM Gateway resolves the connection by email domain, redirects to Entra, and on return the (SCIM-provisioned) user is signed in and lands in the enterprise organization.

## Enforce SSO (optional, do this last) [#enforce-sso-optional-do-this-last]

Flip **Require SSO** on the connection to require SSO for that email domain — password, social, and passkey sign-in are then rejected for it.

<Callout type="warn">
  Only enable **Require SSO** after the test above passes, and remember it also
  applies to the admin setting it up if their email is on that domain — verify
  your own SSO login works first.
</Callout>

## Troubleshooting: login bounces back to the login page [#troubleshooting-login-bounces-back-to-the-login-page]

If the Entra sign-in itself succeeds but the user lands back on the LLM Gateway login page (with a 401 in the browser console), the SAML response was received but LLM Gateway refused to attach it to the provisioned user — almost always an `account_not_linked` domain mismatch:

* **The asserted email's domain isn't in the connection's Email domains list.** Entra sends the user's `mail` attribute as the email; if its domain differs from the one you registered (e.g. you registered the UPN domain), edit the connection's **Email domains** on the SSO page to include both.
* **Wrong Identity provider type.** If the connection was created as &#x2A;*Other (SAML 2.0)** instead of **Microsoft Entra ID**, the Entra claim mapping isn't applied and the email falls back to the NameID (usually the UPN), which may again be off-list — and creates duplicate users instead of matching the SCIM-provisioned ones.

## Troubleshooting: can't find "Enterprise applications" [#troubleshooting-cant-find-enterprise-applications]

* **Insufficient role** — you need Application Administrator, Cloud Application Administrator, or Global Administrator. Ask whoever administers your Microsoft 365 / Entra tenant.
* **Wrong menu** — you're in **App registrations** (for developers) instead of **Enterprise applications**.
* **Wrong tenant** — if you belong to multiple tenants, use **Settings (gear) → Directories + subscriptions** (top-right) to switch to the tenant you administer.


# Google Workspace
URL: https://docs.llmgateway.io/features/sso/google


Google Workspace works differently from the SAML providers: employees already sign in to LLM Gateway with the standard **Sign in with Google** button using their Workspace-managed Google account. Instead of registering a SAML application, you configure an **auto-join email domain** — anyone who signs in with a Google account on that domain is added to your organization automatically.

<Callout type="info">
  Google Workspace auto-join requires the **Enterprise** plan and is configured
  by an organization **owner** or **admin**. See the [SSO
  overview](https://docs.llmgateway.io/features/sso) for the other connection types.
</Callout>

## How it works [#how-it-works]

1. An admin sets the organization's auto-join domain (e.g. `acme.com`) on the **SSO** page.
2. A user signs in (or signs up) with **Sign in with Google** using their `@acme.com` Workspace account.
3. LLM Gateway matches the verified email domain and adds them to the organization with the **developer** role.
4. The user receives an email letting them know they've been added.

This applies to **new signups and existing users** alike — an existing LLM Gateway user on your domain joins the organization on their next Google sign-in. Members are never added twice, and members you've already invited are unaffected.

<Callout type="info">
  Auto-join is **just-in-time provisioning**: users join when they sign in.
  Google Workspace does not support SCIM provisioning to custom apps, so there
  is no directory sync — removing someone from your Workspace does not remove
  them from the organization. Offboard members on the **Team** page.
</Callout>

## Setting it up [#setting-it-up]

<Steps>
  <Step>
    ### Open the SSO page [#open-the-sso-page]

    In the dashboard, go to your organization's **SSO** page and find the **Connections** card.
  </Step>

  <Step>
    ### Add a Google Workspace connection [#add-a-google-workspace-connection]

    Under **Add a connection**, choose **Google Workspace** as the identity provider. There are no SAML URLs or certificates to exchange — only your email domain.
  </Step>

  <Step>
    ### Enter your email domain [#enter-your-email-domain]

    Enter the corporate domain your Workspace accounts use, e.g. `acme.com`, and click **Enable auto-join**.

    * Only **one domain** per organization.
    * A domain can be claimed by **only one organization** — if another organization already uses it, you'll get a conflict error.
    * **Consumer email domains** (`gmail.com`, `outlook.com`, `yahoo.com`, …) are rejected.
  </Step>

  <Step>
    ### Verify with a test account [#verify-with-a-test-account]

    Sign in with a Google account on your domain that isn't yet a member. After signing in, it should appear on the **Team** page with the **developer** role, and receive a notification email.
  </Step>
</Steps>

## Managing the connection [#managing-the-connection]

The Google Workspace connection appears as a card in the **Connections** list:

* **Edit** the domain with the pencil icon — existing members are unaffected; the new domain applies to future sign-ins.
* **Remove** the connection to stop auto-joining. Existing members stay in the organization.

Auto-joined members get the **developer** role and receive the organization's **Default project access** selection, exactly like SSO/SCIM-provisioned members. Promote them or adjust project access on the **Team** page if they need more.

## What's not available with Google Workspace [#whats-not-available-with-google-workspace]

With a Google-only setup, the &#x2A;*Directory sync (SCIM)**, **Group role mapping**, and **Group team mapping** cards on the SSO page are disabled:

* **SCIM** — Google Workspace doesn't support SCIM provisioning for custom apps, so there's no directory sync to configure.
* **Group role mapping** — role mappings apply to groups pushed via SCIM, so they can't take effect either. Auto-joined members always start as **developer**.
* **Group team mapping** — team mappings also rely on SCIM-pushed groups. Assign auto-joined developers to teams on the **Members** page instead.

**Default project access** works as usual and applies to auto-joined members.

## Interaction with SAML and Require SSO [#interaction-with-saml-and-require-sso]

An organization has **one connection** for now — either a SAML connection or Google Workspace auto-join. To switch providers, remove the existing connection first.

For context on why they don't mix: a SAML connection's **Require SSO** toggle blocks password, social, **and Google** sign-in for its domains, so Google sign-in would be rejected before auto-join could run. Pick the model that fits:

* **Google Workspace auto-join** — zero IdP setup, members use the Google button, join on first sign-in.
* **SAML with Require SSO** — stricter control (blocks all non-SSO sign-in), needs a SAML app in your IdP. Google Workspace can also act as a SAML 2.0 IdP via a custom SAML app if you need enforcement; use the &#x2A;*Other (SAML 2.0)** connection type for that.


# SSO
URL: https://docs.llmgateway.io/features/sso


LLM Gateway supports per-organization &#x2A;*SAML 2.0 single sign-on (SSO)** and **SCIM 2.0 directory provisioning**, so an enterprise team can let members sign in with their corporate identity provider and keep membership in sync automatically.

<Callout type="info">
  SSO & SCIM are available on the **Enterprise** plan only, and can be
  configured by an organization **owner** or **admin**. Until the organization
  is on the Enterprise plan, the **SSO** page shows a contact-sales card.
  Contact us at [contact@llmgateway.io](mailto:contact@llmgateway.io) to enable it.
</Callout>

## What each part does [#what-each-part-does]

* **SSO (authentication)** — lets your members sign in with your identity provider (Microsoft Entra ID, Okta, or any SAML 2.0 IdP). Sign-in is routed by **email domain**: a user clicks **Sign in with SSO**, enters their work email, and is redirected to the correct provider. A connection can list **multiple domains** (comma-separated, editable on the connection card) — the list must cover every domain the IdP may assert as a user's email, e.g. both the UPN and `mail` domains on Entra when they differ. A first-time SSO sign-in automatically joins the user to the organization that owns the connection as a **developer** with the configured default project access — it does not create the organization itself.
* **SCIM (provisioning)** — pushes user and group create / update / deactivate / delete from your directory into the organization, so people are added and removed automatically. Provisioned members default to the **developer** role.
* **Group → role mapping*&#x2A; &#x2A;(optional)* — map directory groups to organization roles (`admin` / `developer`). On any membership change, a member receives the **highest** mapped role among their groups. Owners are never auto-demoted.
* **Group → team mapping*&#x2A; &#x2A;(optional)* — map directory groups to organization teams so developers inherit team project, IAM, API-key, and budget policy. Team assignments follow group membership on every directory sync; explicit assignments made by an admin are preserved.
* **Default project access*&#x2A; &#x2A;(optional)* — choose which projects newly provisioned SSO/SCIM members (the `developer` role) can access by default. Owners/admins always see every project, so this only affects developers. Left unconfigured, new members get the organization's first (default) project. Applies to members provisioned after you set it, not retroactively.

## Prerequisites [#prerequisites]

1. **Membership is granted on sign-in.** Anyone your IdP authenticates for the connection joins the organization on their first SSO sign-in (as a **developer**). Members can also join earlier through normal signup/invite or by being pushed via SCIM.
2. **The organization is on the Enterprise plan.** An admin flips the org to `enterprise` to unlock the feature.
3. **`SSO_ENABLED=true` on the deployment.** This global env flag controls whether the **Sign in with SSO** button appears on the login page. The admin config page works without it, but end users can't log in via SSO until it's on.

## The URLs LLM Gateway gives your IdP [#the-urls-llm-gateway-gives-your-idp]

Everything your identity provider needs is derived from the **management API base URL** and a **Connection ID** you choose. On the hosted service the base URL defaults to `https://internal.llmgateway.io` (overridable via `API_URL`); self-hosted deployments use whatever `API_URL` is set to.

<Callout type="warn">
  The management API base URL is **not** `api.llmgateway.io` — that host serves
  the `/v1` chat-completions gateway. SSO/SAML and SCIM live on the management
  API. You don't need to construct these URLs by hand: the exact,
  fully-qualified values are shown with copy buttons on the **SSO** page and on
  each connection card.
</Callout>

With `API_URL = https://internal.llmgateway.io` and a connection slug of `acme-entra`:

| Value                             | URL                                                                               |
| --------------------------------- | --------------------------------------------------------------------------------- |
| Identifier (Entity ID) / Audience | `https://internal.llmgateway.io/auth/sso/saml2/sp/metadata?providerId=acme-entra` |
| Reply URL (ACS)                   | `https://internal.llmgateway.io/auth/sso/saml2/sp/acs/acme-entra`                 |
| SCIM Tenant URL                   | `https://internal.llmgateway.io/scim/v2`                                          |

## SCIM authentication [#scim-authentication]

SCIM uses a **static bearer token**, not OAuth. You generate the token once on the &#x2A;*SSO → Directory sync (SCIM)** card, and your IdP sends it as `Authorization: Bearer <token>` on every request. In your IdP's provisioning config, that means:

* **Authentication method:** the **bearer token*&#x2A; option — Entra labels it &#x2A;*"Long-lived bearer token"*&#x2A; / &#x2A;*"Secret token"**; Okta calls it an **API token**. Do **not** choose an OAuth 2.0 authorization-code flow — LLM Gateway does not implement one, and the connection test will fail.
* **Tenant / base URL:** the SCIM Tenant URL above (`…/scim/v2`).
* **Secret token:** the value from the SCIM card (shown once; rotate to reissue).

The token is scoped to a single organization, so every user and group your IdP pushes lands in that org.

## Provider guides [#provider-guides]

* [Microsoft Entra ID](https://docs.llmgateway.io/features/sso/entra) — set up SAML SSO and SCIM provisioning with Entra ID (formerly Azure AD).
* [Okta](https://docs.llmgateway.io/features/sso/okta) — set up SAML SSO and SCIM provisioning with a custom Okta app integration.
* [Google Workspace](https://docs.llmgateway.io/features/sso/google) — no SAML app needed: members sign in with the standard **Sign in with Google** button and auto-join your organization by email domain.

Any other SAML 2.0 IdP follows the same flow; pick the matching **Identity provider** type when you create the connection so the correct claim mapping is applied.

## End-user sign-in flow [#end-user-sign-in-flow]

1. On the login page the user clicks **Sign in with SSO** (visible when `SSO_ENABLED=true`).
2. They enter their work email, e.g. `jane@acme.com`.
3. LLM Gateway resolves the domain `acme.com` → finds the org's SSO connection → redirects to the org's IdP.
4. The user authenticates at their IdP (usually instant if already signed in).
5. They return signed in, landing in the enterprise organization.

## Enforcing SSO (optional, do this last) [#enforcing-sso-optional-do-this-last]

Each connection has a **Require SSO** toggle. When on, anyone whose email domain matches the connection can *only* sign in via SSO — password, social, and passkey sign-in are all rejected for that domain.

<Callout type="warn">
  Only turn on **Require SSO** after you've confirmed SSO login works
  end-to-end. Enforcing before the IdP works can lock out the whole domain —
  including the admin setting it up if their own email is on that domain.
</Callout>


# Okta
URL: https://docs.llmgateway.io/features/sso/okta


This guide walks an organization admin through connecting **Okta** to LLM Gateway for SAML single sign-on and SCIM provisioning.

<Callout type="info">
  SSO & SCIM require the **Enterprise** plan and an organization **owner** or
  **admin**. See the [SSO overview](https://docs.llmgateway.io/features/sso) for prerequisites and how the
  URLs are derived.
</Callout>

## Create the SAML app integration [#create-the-saml-app-integration]

You'll create a custom SAML 2.0 app integration in the Okta Admin Console to represent LLM Gateway. There is no OIN (Okta Integration Network) catalog listing — the custom app is the supported path, and the steps below cover everything.

<Steps>
  <Step>
    ### Choose a connection slug in LLM Gateway [#choose-a-connection-slug-in-llm-gateway]

    In LLM Gateway (**Organization → SSO → Add a connection**), set **Identity provider** to **Okta**, then pick a **connection slug** (lowercase letters, numbers, hyphens; `^[a-z0-9-]+$`, up to 64 chars). We recommend the format `<your-org-slug>-<provider>`, e.g. `acme-okta` — the form pre-fills this from your organization name and the selected provider. It must be unique across all LLM Gateway organizations, and each organization has exactly one SSO connection. Keep it stable: your SP URLs are built from it.

    As soon as the slug is set, the form shows your two SP URLs (with the hosted default `https://internal.llmgateway.io`) — copy them from there:

    * **Audience URI (SP Entity ID):** `https://internal.llmgateway.io/auth/sso/saml2/sp/metadata?providerId=acme-okta`
    * **Single sign-on URL (ACS):** `https://internal.llmgateway.io/auth/sso/saml2/sp/acs/acme-okta`

    You'll paste these into Okta next — no need to create the connection yet.
  </Step>

  <Step>
    ### Open the Okta Admin Console [#open-the-okta-admin-console]

    Go to your Okta admin console (`https://<your-org>-admin.okta.com`) with an account that has the **Application Administrator** role (or Super Admin). In the left navigation, open **Applications → Applications**.
  </Step>

  <Step>
    ### Create the app integration [#create-the-app-integration]

    Click **Create App Integration**, choose **SAML 2.0** as the sign-in method, and click **Next**. On **General Settings**, set the **App name** (e.g. `LLM Gateway`) and click **Next**.

    <Callout type="warn">
      Pick **SAML 2.0**, not **OIDC**: LLM Gateway's per-organization SSO speaks
      SAML. Don't use **Browse App Catalog** either — there is no gallery listing,
      so always go through **Create App Integration**.
    </Callout>
  </Step>

  <Step>
    ### Configure SAML settings [#configure-saml-settings]

    On the **Configure SAML** step, fill in **Section A – SAML Settings → General**:

    * **Single sign-on URL:*&#x2A; your ACS URL from step 1. Keep &#x2A;*"Use this for Recipient URL and Destination URL"** checked.
    * **Audience URI (SP Entity ID):** your metadata URL from step 1.
    * **Default RelayState:** leave blank.
    * **Name ID format:** **EmailAddress** — required, see below.
    * **Application username:** **Email**.
    * **Update application username on:** Create and update (the default).

    The **Show Advanced Settings** defaults are correct (Response **Signed**, Assertion Signature **Signed**, RSA-SHA256) — LLM Gateway requires signed assertions, which Okta does by default.

    Click **Next**, and on the **Feedback*&#x2A; step select &#x2A;*"This is an internal app that we have created"**, then **Finish**.

    <Callout type="warn">
      **Name ID format = EmailAddress** and **Application username = Email** are
      what make sign-in work: in **Okta** mode, LLM Gateway reads the user's email
      from the SAML **NameID**. You don't need any attribute statements (recent
      versions of the wizard don't even show that section — that's fine); if you do
      add them, `displayName`, `givenName` and `surname` are picked up for the
      user's profile name.
    </Callout>
  </Step>

  <Step>
    ### Grab the Sign-on URL and certificate [#grab-the-sign-on-url-and-certificate]

    After the wizard finishes you land on the app's **Sign On** tab. Under **Settings → Sign on methods → SAML 2.0 → Metadata details**, click **More details** to reveal:

    * **Sign on URL** — looks like `https://<your-org>.okta.com/app/<app-label>/<appId>/sso/saml`, where `<appId>` starts with `exk`. Copy it.
    * **Signing Certificate** — click **Copy** (or **Download**; the file contents are a PEM block `-----BEGIN CERTIFICATE----- … -----END CERTIFICATE-----`).

    <Callout type="warn">
      Use this exact app-specific **Sign on URL** — LLM Gateway derives the SAML
      issuer it expects (`http://www.okta.com/<appId>`) from the `exk…` segment of
      the URL. Pasting some other Okta URL (your org homepage, the metadata URL,
      an Identity-Provider route) makes every SAML response fail issuer
      validation.
    </Callout>
  </Step>

  <Step>
    ### Create the connection in LLM Gateway [#create-the-connection-in-llm-gateway]

    Back on the **SSO → Add a connection** form:

    * **Identity provider:** **Okta** — this selects the email-NameID mapping and issuer derivation described above
    * **Connection slug:** `acme-okta` (must match the slug whose URLs you pasted into Okta)
    * **Email domains:** your corporate domain(s), e.g. `acme.com` (comma-separate for several)
    * **Identity Provider Single Sign-On URL:** the Okta **Sign on URL**
    * **X.509 signing certificate:** paste the PEM block

    Save. The connection card then shows the exact SP metadata + ACS URLs — confirm they match what you entered in Okta. You can edit the email domain list on the connection card later without re-creating the connection.

    <Callout type="warn">
      The domain list must include **every domain Okta may send as a user's email**.
      With **Application username = Email**, that's the domain of each user's Okta
      email address. If some users sign in with a different domain (e.g. a
      subsidiary), list all of them — otherwise those users bounce back to the login
      page with an `account_not_linked` error even though Okta authenticated them.
    </Callout>
  </Step>
</Steps>

## Assign users to the app [#assign-users-to-the-app]

Configuring SAML isn't enough on its own — Okta only lets **assigned** users sign in to the app. On the app's **Assignments** tab, click **Assign → Assign to People** (or **Assign to Groups**), add the people who should be able to sign in, and confirm. The **Applications** page's **Assign Users to App** bulk flow works too.

<Callout type="warn">
  If you skip this, sign-in fails at Okta with &#x2A;"Sorry, you can't access LLM
  Gateway because you are not assigned this application in Okta."* — even though
  the SAML config is correct. Assign at least your own account before testing.
</Callout>

If you also set up SCIM below, assignment doubles as the provisioning trigger: assigned users are pushed to LLM Gateway.

## Configure SCIM provisioning [#configure-scim-provisioning]

SSO only signs people in. SCIM is what actually adds members to the organization. Set it up on the **same** app integration.

<Steps>
  <Step>
    ### Generate a SCIM token [#generate-a-scim-token]

    In LLM Gateway, on &#x2A;*Organization → SSO → Directory sync (SCIM)**, click **Generate SCIM token** and copy it (shown once). The SCIM base URL is shown in the same card — it's `https://internal.llmgateway.io/scim/v2` on the hosted default.
  </Step>

  <Step>
    ### Enable SCIM on the app [#enable-scim-on-the-app]

    On the app's **General** tab → **App Settings** → **Edit**, set **Provisioning** to **SCIM** and **Save**. A new **Provisioning** tab appears on the app.
  </Step>

  <Step>
    ### Configure the SCIM connection [#configure-the-scim-connection]

    On **Provisioning → Integration** → **Edit**:

    * **SCIM connector base URL:** the SCIM base URL, e.g. `https://internal.llmgateway.io/scim/v2`
    * **Unique identifier field for users:** `userName`
    * **Supported provisioning actions:** check **Push New Users**, **Push Profile Updates**, and **Push Groups** (leave the two *Import* options unchecked — LLM Gateway doesn't support importing users into Okta)
    * **Authentication Mode:** **HTTP Header** — Okta then shows an **Authorization: Bearer** field; paste the SCIM token there. **Do not** pick Basic Auth or OAuth 2: LLM Gateway's SCIM endpoint authenticates a plain static bearer token.

    Click **Test Connector Configuration*&#x2A; — a green &#x2A;*"Connector configured successfully"** panel should list **Create Users**, **Update User Attributes** and **Push Groups** as detected (the *Import* rows show a red ✗, which is expected). Then **Save**.
  </Step>

  <Step>
    ### Enable provisioning to the app [#enable-provisioning-to-the-app]

    On **Provisioning → To App** → **Edit**, enable **Create Users**, **Update User Attributes**, and **Deactivate Users**, then **Save**. Leave **Sync Password** off.

    From now on, assigning a user (or an assigned group's member) pushes them to LLM Gateway; unassigning or deactivating them removes their membership.

    <Callout type="warn">
      Users who were **assigned before provisioning was enabled** are not pushed
      automatically — they show a red warning icon on the **Assignments** tab. Click
      the **Provision User** button there and confirm with **OK**; Okta runs a job
      that provisions all such users.
    </Callout>
  </Step>
</Steps>

Provisioned users are created in LLM Gateway and added to the organization (default role **developer**). Deactivating or unassigning a user in Okta removes their membership.

## Group → role mapping (optional) [#group--role-mapping-optional]

On the app's **Push Groups** tab, push the Okta groups whose membership should matter. Then on the LLM Gateway **SSO** page, add mappings such as `Acme Admins → admin`. When Okta pushes group membership, each member gets the **highest** mapped role among their groups (default `developer`). Owners are never auto-demoted, so you can't accidentally lock yourself out of the owner role.

## Group → team mapping (optional) [#group--team-mapping-optional]

Push each relevant group from the app's **Push Groups** tab, then use the **Group team mapping** card on the LLM Gateway **SSO** page to map its exact name to an organization team. Developer members inherit that team's project, IAM, API-key, and budget policy.

* Assignments are recomputed on provisioning and reactivation, group membership pushes, renames and deletions, and immediately when a mapping is saved or deleted.
* Explicit team assignments made by an admin are preserved. Removing only a synced assignment is temporary; the next SCIM event restores the mapping.
* A developer can belong to one team. If several mapped groups apply, the alphabetically first exact group name wins. Owners and admins cannot belong to teams, so a privileged role mapping takes precedence.
* Team project access intersects with the member's personal/default project grants; both must allow a project.

<Callout type="warn">
  If the mapped group is also the user's only assignment to the Okta app,
  removing them can deprovision the user and revoke their API keys. Assign users
  through a broader group or directly when group leave should only change team
  policy.
</Callout>

## Default project access (optional) [#default-project-access-optional]

Provisioned members join as **developers**, who only see the projects they're granted (owners/admins always see every project). Use the **Default project access** card on the **SSO** page to choose which projects newly provisioned SSO/SCIM developers get by default. If you leave it unconfigured, new members get your organization's first (default) project so they can see something out of the box; select an empty set to instead grant nothing and manage project access manually per member.

<Callout type="info">
  Default project access applies only to members provisioned **after** you set
  it — it doesn't retroactively change existing members' project grants. Adjust
  an existing member's projects from the **Team** page.
</Callout>

## Test the login [#test-the-login]

Log out → login page → **Sign in with SSO** → enter a work email (`someone@acme.com`). LLM Gateway resolves the connection by email domain and redirects to Okta; if the user is already signed in to Okta the round-trip is instant. On return they're signed in as the (SCIM-provisioned) user — if the dashboard opens in the user's own personal organization, the enterprise organization is one click away in the organization switcher.

## Enforce SSO (optional, do this last) [#enforce-sso-optional-do-this-last]

Flip **Require SSO** on the connection to require SSO for that email domain — password, social, and passkey sign-in are then rejected for it.

<Callout type="warn">
  Only enable **Require SSO** after the test above passes, and remember it also
  applies to the admin setting it up if their email is on that domain — verify
  your own SSO login works first.
</Callout>

## Troubleshooting [#troubleshooting]

* **Okta says "you are not assigned this application"** — the SAML config is fine; the user just isn't on the app's **Assignments** tab. Assign them (or their group).
* **Okta sign-in succeeds but the user lands back on the LLM Gateway login page** — the SAML response was received but refused, almost always an `account_not_linked` domain mismatch: the asserted email's domain isn't in the connection's **Email domains** list. Edit the list on the connection card to cover every domain Okta may assert.
* **Every SSO attempt fails after the Okta redirect** — check that the connection's **Identity Provider Single Sign-On URL** is the app-specific **Sign on URL** containing the `exk…` app id (Sign On tab → Metadata details → More details). Other URLs break issuer validation. Also confirm the connection was created with **Identity provider = Okta** and that the certificate is the app's current **SHA-2 Active** signing certificate.
* **Assigned users never appear in the organization** — provisioning was probably enabled after they were assigned. Use **Provision User** on the **Assignments** tab, and confirm **Provisioning → To App → Create Users** is enabled and the connector test passes.