OpenAI-Compatible Batch API

Running an eval set, an enrichment job or a backfill through the gateway meant writing your own queue: cap the concurrency, back off on 429, resume after a crash, and keep track of which lines failed. The new Batch API does that for you. Upload a JSONL file, create a batch, and download the results when it finishes.

It follows OpenAI's Batch API, so the OpenAI SDKs work once the base URL points at the gateway.

From file to results

Each line of the input file is one request. Its custom_id comes back on the result, and body is the same JSON you would send to the endpoint directly:

1{"custom_id": "review-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize: ..."}]}}2{"custom_id": "review-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize: ..."}]}}

1{"custom_id": "review-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize: ..."}]}}2{"custom_id": "review-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarize: ..."}]}}

Upload it, then create the batch:

1curl https://api.llmgateway.io/v1/files \2  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \3  -F purpose=batch \4  -F file=@requests.jsonl5
6curl https://api.llmgateway.io/v1/batches \7  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \8  -H "Content-Type: application/json" \9  -d '{10    "input_file_id": "file-abc123",11    "endpoint": "/v1/chat/completions",12    "completion_window": "24h"13  }'

1curl https://api.llmgateway.io/v1/files \2  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \3  -F purpose=batch \4  -F file=@requests.jsonl5
6curl https://api.llmgateway.io/v1/batches \7  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \8  -H "Content-Type: application/json" \9  -d '{10    "input_file_id": "file-abc123",11    "endpoint": "/v1/chat/completions",12    "completion_window": "24h"13  }'

Poll the batch. An input file with invalid lines ends the batch as failed, with up to 100 problems and their line numbers in errors and no result files. Otherwise, once it is completed, expired or cancelled, download the files it produced:

1import os2
3from openai import OpenAI4
5client = OpenAI(6    base_url="https://api.llmgateway.io/v1",7    api_key=os.environ["LLM_GATEWAY_API_KEY"],8)9
10batch = client.batches.retrieve("batch_abc123")11print(batch.status, batch.request_counts)12
13if batch.status == "failed":14    print(batch.errors)15elif batch.status in ("completed", "expired", "cancelled"):16    if batch.output_file_id:17        output = client.files.content(batch.output_file_id).text18    if batch.error_file_id:19        errors = client.files.content(batch.error_file_id).text

1import os2
3from openai import OpenAI4
5client = OpenAI(6    base_url="https://api.llmgateway.io/v1",7    api_key=os.environ["LLM_GATEWAY_API_KEY"],8)9
10batch = client.batches.retrieve("batch_abc123")11print(batch.status, batch.request_counts)12
13if batch.status == "failed":14    print(batch.errors)15elif batch.status in ("completed", "expired", "cancelled"):16    if batch.output_file_id:17        output = client.files.content(batch.output_file_id).text18    if batch.error_file_id:19        errors = client.files.content(batch.error_file_id).text

Requests that returned a 2xx response land in the output file. The error file holds the rest: non-2xx responses, requests that had not run when the batch was cancelled or its window ended, and responses over 10 MB, which are not stored and fail with response_too_large. A batch has each file only when it has lines for it, and both keep the input order.

Three endpoints, any model they accept

The model is part of each line's body, so one batch can send different lines to different models. Every line runs through the normal gateway path: routing, fallback, caching, IAM rules, guardrails and usage limits apply, and each request shows up in your logs with the source llmgateway-batch. Streaming requests are not supported, and neither are realtime, image-generation and video models or custom models that only stream. The models page marks every model that works in a batch.

What gets retried

• Rate limits and empty balances wait. A 429, an overloaded gateway (529), insufficient credits (402), or an API key or member budget at its spend limit does not fail a request. It waits and retries until the window ends, so a batch resumes after you top up.
• Provider outages are retried. 502, 503, 504, a 500 with code upstream_error and connections that never reached the gateway are attempted up to three times in total. These errors are not billed, so a retry never charges twice.
• A request with an unknown outcome is not retried. If the connection drops mid-request or the request times out, the line fails with request_failed; if the worker restarts while it runs, it fails with request_interrupted. Such a line may still have been billed, but its result is lost.

Same price as a live request

Each line is billed at the same per-token price as the same request sent live, and there is no fee for storing batch files. Discounts that providers offer on their own batch APIs are not passed through, because the gateway runs each line as a regular request. The Batch API saves you the queueing code, not money.

Limits

Batch files hold your prompts and completions. They are kept for 30 days, or until you delete them, regardless of your data retention setting. While zero data retention is active, uploading files and creating batches return 400 with code zdr_storage_conflict.

Batch API docs → | Models with batch support →