Thousands of requests, one upload

Batch API

Upload a JSONL file of up to 50,000 requests and download the results when the batch finishes, through the OpenAI-compatible /v1/batches API.

Get startedRead the docs
batch_abc123 · completed
Live
  • requests.jsonl uploaded50,000 lines
  • Rate limited, waited429
  • Provider outage, retriedattempt 2
  • Output file ready49,987 done
01

OpenAI format

client.batches works unchanged.

02

No queue to build

Concurrency, pacing and retries handled.

03

Never billed twice

Only unbilled errors are retried.

04

A model per line

Chat, responses and embeddings.

Built for

  • Evals

    Score a test set overnight

  • Enrichment

    Classify or summarize a table

  • Backfills

    Embed an archive in one job

Create a batch
terminal
curl https://api.llmgateway.io/v1/batches \  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "input_file_id": "file-abc123",    "endpoint": "/v1/chat/completions",    "completion_window": "24h"  }'
Read docs

More in routing & reliability

All features

Smart Routing & Fallback

The best provider, every request

Dynamic Routes

Routing logic, out of your code

Sessions

Sticky routing for whole conversations

Start using Batch API today

One API key. Every model. Live in minutes.

Start buildingBrowse models