AI & LLM glossary
Short, plain-language definitions of the terms you meet when building with LLM APIs.
AI gateway
An AI gateway is a service that sits between your application and AI model providers. It exposes one API and one key, forwards each request to the right provider, and adds the controls a single provider does not: failover, cost tracking, rate limits, caching, and access policies across every model you use.
Bring your own key (BYOK)
Bring your own key (BYOK) means using your own provider API keys through a gateway instead of buying usage from the gateway. Requests are billed by the provider on your existing account and agreements, while the gateway still supplies one API, routing, logging, and analytics across all of your keys.
Context window
A context window is the maximum number of tokens a model can process in one request, counting the prompt, the conversation history, any documents or tool definitions, and the response it generates. Text beyond the limit is rejected or must be removed, so the context window sets how much a model can consider at once.
Embeddings
An embedding is a list of numbers that represents the meaning of a piece of text. Texts with similar meaning produce vectors that are close together, so embeddings let software compare and search content by meaning instead of exact words. They are the basis of semantic search and retrieval-augmented generation.
Guardrails
Guardrails are checks applied to AI requests and responses to block or change content that breaks a policy. Typical rules detect personal data, secrets, prompt injection, and unsafe topics. Running them in a gateway applies the same rules to every model and application instead of rebuilding them in each codebase.
LLM routing
LLM routing is the choice of which provider, and sometimes which model, handles each request. A router weighs price, latency, uptime, and the features a request needs, then sends it to the best available option. Routing lets one application use many providers without hard-coding any of them.
Prompt caching
Prompt caching lets a provider store the beginning of a prompt after the first request and reuse it on later requests that start with the same text. Cached input tokens are billed at a lower rate than normal input tokens and are processed faster, which cuts cost and latency for long, repeated prompts.
Provider fallback
Provider fallback is retrying a failed AI request on a different provider that serves the same model. When the first provider returns a server error, a rate limit, or a timeout, the request is sent to the next one inside the same API call, so the application receives an answer instead of an error.
Rate limit
A rate limit is a cap on how much you can use an API in a period of time, usually counted in requests per minute or tokens per minute. When you exceed it the API returns HTTP 429 and refuses the request. Rate limits protect providers from overload and spread capacity across customers.
Reasoning tokens
Reasoning tokens are tokens a model generates while working through a problem before it writes its final answer. They are billed as output tokens even when the provider does not show the reasoning text. Higher reasoning effort usually improves hard tasks and raises both cost and response time.
Streaming
Streaming is delivering a model's response piece by piece as it is generated instead of waiting for the full answer. The API keeps the connection open and sends small chunks, typically as server-sent events. Users see text appear immediately, which makes long answers feel fast even when total generation time is unchanged.
Structured outputs
Structured outputs make a model return data in a format your code can parse, usually JSON that matches a schema you supply. With strict schema enforcement the provider guarantees the response is valid against the schema. Without it, the model is only instructed to produce JSON and can still return something malformed.
Time to first token (TTFT)
Time to first token (TTFT) is the delay between sending a request and receiving the first piece of the model's response. It measures how responsive a streaming model feels. TTFT depends on the model, the provider serving it, prompt length, and current load, and it is separate from how fast tokens arrive afterward.
Token
A token is the unit of text a language model reads and writes. Models split text into tokens, which are often word fragments: one token is roughly four characters of English. Providers price requests per token and limit how many tokens a request may contain, so token counts drive both cost and context limits.
Tool calling
Tool calling lets a model ask your application to run a function instead of answering in text. You describe the available tools and their parameters; the model returns the tool name and arguments as structured data; your code runs the tool and sends the result back so the model can continue.
Zero data retention (ZDR)
Zero data retention (ZDR) means a service does not store the content of your prompts and responses after it has processed them. Under ZDR a provider keeps no request payloads for logging, abuse review, or training. Teams handling sensitive data use it to limit where that data can persist.