The LLM Compliance Checklist for Production Teams

Security reviews for LLM features fail on the same handful of questions. Which providers see our data? Can anything route somewhere we haven't approved? What do we retain, and what do they retain? Who approved this model, and where's the record?

This LLM compliance checklist is the eight-point version of that review — the one to run before the vendor questionnaire arrives. For each point: what good looks like, and whether it's something you can enforce mechanically at the gateway or something you document by hand. Enforcement beats documentation every time an outage, a retry, or a hurried engineer would otherwise make the decision for you.

LLM Gateway enforces most of this list at the routing layer; where a capability is Enterprise-gated, we say so.

1. Inventory every provider that can see your data

You can't review what you can't list. Every provider your traffic can reach — including fallbacks — receives your data, typically as a processor (confirm the exact role — processor, subprocessor, or controller — in each provider's DPA), and the list is longer than most teams think once automatic failover is involved.

• What good looks like: a live list of eligible providers per model, not a wiki page from the launch review.
• How to enforce it: route everything through one gateway so there is exactly one place the list lives. The providers directory shows every catalogue provider with its certifications, data policy, and headquarters.

2. Restrict routing to providers that meet your requirements

An approved-vendor list only matters if unapproved vendors are unreachable.

• What good looks like: requests physically cannot reach a provider that fails your certification or data-policy bar, on any code path.
• How to enforce it: provider compliance policies (Enterprise plan) — toggle SOC 2 Type II, ISO 27001, GDPR, no-training-on-prompts, no-prompt-logging, and no-stealth-providers requirements. Every requirement is fail-closed: unknown attributes count as non-compliant, and blocked requests get a 403 before any data leaves the gateway. The dashboard's policy-aware provider pickers show a green or red shield per provider, with the exact failing requirements listed, so the policy is visible while you configure it — not just when it rejects traffic.

3. Control where data goes geographically

Certifications tell you how a provider behaves; jurisdiction tells you which laws it answers to.

• What good looks like: routing restricted to approved countries, with regional pinning where a provider offers multiple regions.
• How to enforce it: the Provider Headquarters filter allows only providers based in your selected countries (fail-closed for unknown headquarters), and provider/model:region syntax pins individual requests to a specific region. See GDPR-compliant LLM routing for the EU-specific walkthrough.

4. Set an explicit retention policy — yours and theirs

Two retention questions, often conflated: what you store, and what your providers store.

• What good looks like: metadata-only storage unless payload retention has a named justification; provider-side logging and training excluded by policy (point 2).
• How to enforce it: LLM Gateway defaults to metadata-only — no prompts or responses stored. Payload retention is per-organization opt-in at $0.01/1M tokens with automatic deletion after the retention period (custom periods on Enterprise). The LLM data retention guide covers the trade-offs.

5. Gate who can use what

Model access is a permission like any other. A developer experimenting with an unapproved model shouldn't be a policy discussion after the fact.

• What good looks like: org-wide restrictions that member- and key-level rules cannot override.
• How to enforce it: blocked/allowed provider and model lists compose with the requirements above, and organization-level compliance restrictions take precedence over member-level and API-key-level IAM rules. Project-scoped developers can browse what they're allowed to call on the org's Models page without being granted policy access.

6. Keep an audit trail of decisions and rejections

Auditors ask two things: show me the control, and show me it working.

• What good looks like: config changes, key rotations, and blocked requests all recorded with actor and timestamp.
• How to enforce it: every compliance-policy block is recorded as a security event, and audit logs (Enterprise) capture configuration and key changes — exportable as evidence for your own SOC 2 or HIPAA audits.

7. Cover your own deployments, not just catalogue providers

Custom deployments — models in your own cloud account — have no published data policy, so a fail-closed policy blocks them by default.

• What good looks like: internal deployments held to the same written standard as external vendors.
• How to enforce it: record a compliance self-attestation per custom provider key (SOC 2 status, logging, training, operating country). The gateway evaluates attestations with the same fail-closed rules, and every attestation change lands in the audit log.

8. Verify your gateway vendor, too

The gateway sees everything, so it has to clear a higher bar than the providers behind it.

• What good looks like: a current SOC 2 Type II report, a public trust center, and — ideally — source you can read.
• How it works here: LLM Gateway holds a SOC 2 Type II report, the trust center at security.llmgateway.io hosts certifications and subprocessors, and the core is AGPLv3 — self-hostable when policy requires the gateway inside your own boundary.

The checklist, in one place

Frequently Asked Questions

What should an LLM compliance checklist cover?

Provider inventory, routing restrictions, geographic control, retention (yours and your providers'), access gating, an audit trail, coverage for internal deployments, and diligence on the gateway vendor itself. The distinguishing question for each: is it enforced mechanically, or documented and hoped for?

Can compliance rules be enforced before data reaches a provider?

Yes — that's the point of enforcing at the gateway. LLM Gateway blocks requests that would violate the compliance policy with a 403 before any data leaves the gateway, on both automatic routing and pinned-provider requests.

Which parts of this need an Enterprise plan?

Provider compliance policies, the headquarters filter, org-level restrictions over IAM, custom retention periods, and audit logs are Enterprise features. Metadata-only retention, the provider directory, and self-hosting the AGPLv3 core are available to everyone.

Does self-hosting remove the need for this checklist?

No — self-hosting moves the gateway inside your boundary but your traffic still fans out to providers. Points 1–5 apply identically; you're just running the enforcement point yourself.