LLM Production Readiness Checklist

Model APIs fail in ways ordinary HTTP APIs do not: they succeed and return the wrong shape, they succeed and stop mid-sentence, they refuse.

• Every response has its stop reason checked, and a truncated answer is treated as a failure rather than an answer.critical

  A response cut off at the token limit arrives with HTTP 200. Nothing about the transport tells you the answer is half-finished — only the stop reason does. This is the single most common silent failure in production LLM systems.

  Check this with the max tokens planner →
• Retries use exponential backoff with jitter and a maximum attempt count.critical

  A rate limit hit by twenty workers that all retry after exactly one second is a rate limit hit by twenty workers one second later. Jitter is what breaks the synchronisation; without it a brief overage becomes a sustained outage.

• Only retryable errors are retried. A 400 and a content refusal are not retried.

  Retrying a malformed request burns quota and latency to receive the same error four more times. Classify before you retry: 429 and 5xx yes, 400 and 401 never.

• Every request has a timeout shorter than the caller is willing to wait.

  Long generations can run for minutes. Without a client timeout a single slow request holds a connection, a worker and a user for as long as the provider is willing to keep talking.

• A response that fails to parse has a defined path — repair, retry once, or fall back — and never throws into the user path.

  Structured output is a strong constraint, not a guarantee. Something eventually comes back with a trailing comma or a markdown fence around it, and the handler for that case should exist before it happens.

  Check this with the llm output json validator →
• There is a defined behaviour when the provider is fully down, and it is not a stack trace.

  Providers have multi-hour incidents. Decide now whether the feature disappears, falls back to a second provider, or serves a cached answer — the decision is worse under pressure.

• A per-user and per-account spend ceiling exists and is enforced in code, not by a billing alert.critical

  An alert tells you after the money is gone. A ceiling stops it. The gap between the two is the entire cost of an abuse incident.

  Check this with the llm cost calculator →
• max_tokens is set on every call to a value based on the longest legitimate answer.

  Left unset, output length is bounded only by the model. One prompt that induces a rambling answer costs the same as thousands of normal ones, and output tokens are the expensive side.

  Check this with the max tokens planner →
• User-supplied input is truncated to a known token ceiling before it reaches the model.critical

  Anyone who can paste can paste a novel. Input length is the one cost variable a user controls directly.

  Check this with the token counter →
• Cost per request has been measured on real traffic, not estimated from a sample prompt.

  Real prompts carry system instructions, retrieved context, conversation history and tool schemas. The estimate from a bare user message is routinely off by an order of magnitude.

  Check this with the llm cost calculator →
• The stable prefix of the prompt is actually stable, and prompt caching is switched on.

  A timestamp or a request id in the system prompt invalidates the cache on every call. The saving is large enough — cached reads cost a fraction of base input — that it is worth checking rather than assuming.

  Check this with the prompt cache checker →
• Token usage is logged per request with enough dimensions to answer "what got expensive" later.

  Aggregate spend tells you the bill went up. Per-request usage with a feature or endpoint tag tells you which change did it.

• An evaluation set of at least fifty real cases exists, with expected outcomes, and is version controlled.critical

  Without one, every prompt change is evaluated by trying three examples that came to mind and declaring victory. Fifty cases is where changes stop being coin flips.

  Check this with the eval dataset builder →
• The evaluation set includes the cases that failed in testing, not only the cases that worked.

  An eval set built from successes measures nothing. The failures are the only part that can regress.

• A baseline score is recorded, and prompt changes are compared to it rather than to intuition.

  Prompt edits routinely improve the case you were looking at and break two you were not. Only a fixed set catches that.

  Check this with the prompt ab significance →
• Prompts live in version control, and a change to a prompt goes through the same review as a change to code.

  A prompt is program logic written in English. Editing it in a dashboard means production behaviour changes with no diff, no review and no way back.

• Temperature is set deliberately, and anything that must be reproducible runs at zero.

  The default is not neutral, it is a choice someone else made. Classification and extraction want determinism; drafting does not.

  Check this with the temperature simulator →

• Untrusted text — retrieved documents, user uploads, web pages — is delimited and marked as data, never concatenated into instructions.critical

  Any text the model reads can attempt to instruct it. Delimiting does not make injection impossible, but concatenating without delimiters makes it trivial.

  Check this with the prompt injection scanner →
• You know exactly which personal data reaches the provider, and it is the minimum the feature needs.critical

  This is the question a data protection review asks first, and the answer "the whole record, it was easier" fails it.

  Check this with the pii redactor →
• A data processing agreement is in place with the provider, and the retention setting matches what your privacy policy says.

  Providers differ on retention defaults and on whether inputs may be used for training. Your policy makes a promise; the setting is what keeps it.

• No prompt contains a credential, an API key or an internal hostname.

  Prompts get logged, cached, and shown in error messages. Anything in one should be safe to read.

  Check this with the secret scanner →
• Model output is never passed to anything that executes it — a shell, a query, a template — without validation.critical

  Model output is untrusted input wearing a helpful tone. The rules that apply to a form field apply here.

• Users can tell they are talking to a model, and the interface does not imply a human is reading.

  Beyond the growing regulatory expectation, it sets the accuracy expectation correctly and makes the occasional wrong answer survivable.

• The feature can be turned off without a deploy.critical

  When a prompt change or a provider incident starts producing bad answers, the time to fix is the time to disable — not the time to review, build and ship a revert.

• The model version is pinned explicitly, not requested through a floating alias.

  A floating alias means the model under your prompt changes on the provider's schedule, not yours, and the first sign is a quality regression nobody deployed.

• The previous prompt and model version can be restored in minutes.

  Rolling back to "whatever it was last Tuesday" is only possible if last Tuesday was recorded.

• Latency is monitored at p95, not at the mean, and there is a number that counts as too slow.

  Generation latency has a long tail by nature. The mean stays comfortable while a quarter of users wait twice as long.

  Check this with the streaming simulator →
• Users can report a bad answer, and those reports reach the eval set.

  This is the cheapest source of real failure cases that exists, and it closes the loop between production and the thing you test against.

• A single user cannot exceed a defined request rate.

  Without it your quota is shared on a first-come basis, and one script takes it all.