LLMOps: how to run LLM apps and AI agents in production

LLMOps is the set of practices for running applications built on large language models in production: versioning prompts and configuration, testing every change against an evaluation set, guarding inputs and outputs, tracing each run, holding cost and latency to a budget, managing model upgrades and deprecations, responding to incidents and protecting the data that passes through. It borrows the discipline of MLOps, but the thing being operated is different. Most teams do not train the model; they rent it, and wrap it in prompts, retrieval, tools and rules that change every week. This guide explains how LLMOps differs from MLOps, walks the lifecycle step by step, gives you a checklist to copy, and sets out who should own what.

LLMOps vs MLOps

Google's MLOps guide describes MLOps as an engineering culture and practice that unifies building an ML system with operating it, and grades teams from level 0, where every step is manual, to level 2, where a CI/CD system builds, tests and deploys pipeline changes. That idea carries over intact: nothing reaches production without an automated test, and every change can be traced and reversed. What changes is the object you are testing.

If you already run MLOps, keep the pipeline and the habits. Add the parts in the right-hand column. If you have never run either, LLMOps is the easier place to start, because there is no training pipeline to build.

The LLMOps lifecycle

Eight areas, in roughly the order a team meets them. Each one links to a longer post where we have written one.

1. Prompt and configuration versioning

A prompt is code. So is the model ID, the temperature or effort setting, the tool schema, the retrieval settings and the guardrail rules. Keep them in version control or a prompt registry, give every combination a version, and stamp that version on every trace. When quality drops, the first question is always "what changed?", and you can only answer it if each run records what it ran on. Never edit a production prompt in a vendor console with no record. Treat a prompt change like a code change: a review, an eval run and a way back.

2. Evaluation sets and regression evals

An eval set is a fixed collection of real inputs with agreed correct outcomes. Run it before launch and on every change to prompt, model, tools or retrieval, and block the deploy when the score falls below the threshold. This is the single practice that separates operating an LLM system from hoping about one. We cover the parts in three posts: how to build an eval set from real tickets, which metrics should gate a deploy, and how to calibrate an LLM-as-judge before you trust it to score live runs. Add every production failure you find to the set, so the same bug cannot ship twice.

3. Guardrails

Guardrails are the checks outside the model that hold whatever the model decides: input filters, output validation, permission limits on tools, and a human approval step where a mistake is costly. The OWASP Top 10 for LLM Applications 2025 is the most useful list to design against. It runs from LLM01 Prompt Injection through Sensitive Information Disclosure, Excessive Agency and System Prompt Leakage to LLM10 Unbounded Consumption. For agents that can act, read our AI agent security framework and where to put people in the loop.

4. Observability and tracing

A wrong answer returns HTTP 200 like a right one, so uptime monitoring tells you nothing about quality. You need a trace for every run: each model call, tool call and retrieval step with inputs, outputs, latency, tokens, cost, version and any score or user feedback. This is what "AI model monitoring" means for an LLM system. Our comparison of AI agent observability tools lists what to capture and compares Langfuse, LangSmith, Phoenix, Helicone, Braintrust, Weave, Datadog and the OpenTelemetry GenAI conventions, with prices. Pick one, instrument from day one, and sample live runs for scoring every week.

5. Cost and latency budgets

OWASP lists unbounded consumption as a top-ten risk for a reason: a retry loop or an agent that keeps calling tools can burn a month's budget in a weekend. Set budgets per run, not just per month.

• Per-run caps. A maximum number of model calls, tool calls and tokens, and a wall-clock timeout. When a run hits a cap, it stops and hands over to a person.
• Cost per outcome. Track cost per resolved case or per processed document, not only total spend. It is the number that tells you whether the system still pays for itself.
• Latency targets. Set them at the slow end, not the average, because users feel the slow runs.
• Routing and caching. Send easy steps to a smaller model and reuse repeated context. Providers price these differently; Anthropic, for example, lists Batch API requests at 50% off, checked on 24 October 2026.
• Alerts. A daily cost alert at a level someone will act on, and a hard spending limit at the provider.

6. Model upgrades and deprecations

Every model you call will be retired. Providers publish their notice periods, and they are measured in months, not years. Here is what each one says on its own page, read on 24 October 2026:

It is not only models. Anthropic's API version 2026-10-21 removed the temperature, top_p and top_k parameters; on that version, a request that sets them returns a 400 error. A system that worked yesterday can fail on a parameter, not a model. What to do about it:

• Pin a specific model version in configuration, never a floating "latest" alias where the provider offers one. Anthropic's docs say every Claude model ID is now a pinned snapshot; other providers differ, so check.
• Keep an inventory of every model, in every environment, with its provider status and retirement date. Check it monthly.
• Run the eval set on the replacement as soon as one is announced, not in the last week. A new model can be better on average and worse on your hardest cases.
• Migrate through a canary. Send a small share of traffic to the new model, compare scores, cost and latency, then switch.
• Watch the cloud you buy through. If you use a model through a cloud marketplace, its retirement dates may not match the model maker's.

7. Incident response

LLM incidents rarely look like outages. They look like a rise in escalations, a customer screenshot of a wrong answer, or a cost spike. Write the response down before you need it: who is on call, what the alert thresholds are (including one for quality), how to read a trace, what to roll back and to what, and who tells the business. Our AI operations runbook is the template. Keep a kill switch that turns the AI path off and routes everything to people, and test that it works. Why AI systems break after launch covers the slow failures that never trip an alert at all.

8. Data and privacy

Prompts and traces hold customer data word for word. Decide what may go to a model provider at all, redact what does not need to, check the provider's terms on training and retention for the account you use, set trace retention to the shortest period your audits allow, and restrict who can read traces. For a framework to hang this on, NIST's Generative AI Profile (NIST AI 600-1), published on 26 July 2024, applies its AI Risk Management Framework to generative AI. For health data, start with HIPAA-compliant AI agents; for staff use of AI tools, our AI policy template.

The LLMOps checklist

Copy this into your tracker. Each line is a yes or no; anything that is a no has an owner and a date.

Before launch

1. A written statement of what the system does and must never do.
2. Prompts, model IDs, tool schemas and retrieval settings in version control, with a version stamped on every run.
3. A specific model version pinned in configuration for every environment.
4. An eval set built from real cases, including the awkward ones, with a pass threshold agreed by the business owner.
5. Evals run automatically on every change, and a failed run blocks the deploy.
6. Input and output guardrails, tool permissions limited to the task, and human approval where a mistake is costly.
7. A trace for every run with inputs, outputs, steps, latency, tokens, cost and version.
8. Per-run caps on calls, tokens and time, and a hard spending limit at the provider.
9. Personal data redacted or kept out of prompts where it is not needed, and trace retention set.
10. A runbook, an on-call owner, alert thresholds and a tested kill switch.

Every week

1. Score a sample of live runs, by people or a calibrated judge.
2. Review escalations, user feedback and the slowest and most expensive runs.
3. Add new failure cases to the eval set.
4. Check cost per outcome against the budget.

Every month

1. Check every model in the inventory against the provider's deprecation page.
2. Re-run the full eval set, even with no changes, to catch silent drift.
3. Review access to traces and production keys.
4. Report quality, cost and volume against the KPIs the system was built to move.

On every model or prompt change

1. Run the eval set and compare against the current version.
2. Canary the change on a share of traffic before switching fully.
3. Keep the previous version ready to restore.
4. Record what changed, why, and the scores before and after.

Roles: who owns what

Small teams combine these roles; one engineer can hold three of them. What matters is that each has a name next to it before launch.

The most common gap is the system owner. Engineers can keep a system running, but someone on the business side has to decide what a correct answer is and whether the system is still worth its cost. When ownership changes hands, from a vendor to your team or between vendors, use our AI system handover checklist so the eval set and runbook move with it.

When you need less than all this

A single internal tool used by five people, with a person reading every output, does not need a canary process or an on-call rota. It still needs three things: a pinned model version, a small eval set you run when you change anything, and a calendar reminder to check the provider's deprecation page. Scale the rest of the checklist with how many people rely on the output and how much a wrong one costs.

Where Gigabit fits

• Run it for you. Managed AI Operations is a monthly retainer: $3,000/mo for one production system, $8,000/mo for three, $20,000/mo for five or more or an agent platform. It covers monitoring, regression evals, prompt and model versioning, drift and response when a model vendor changes something, and we take over systems your team or another vendor built, starting with an audit. In three published cases, operations automation for a retailer, a support agent and AI across a consulting firm, the monthly operating fee was $2,500, $3,500 and $4,500.
• Build the LLMOps setup into your team. Embedded AI Teams places engineers, including MLOps engineers, inside your team at $3,000–$7,500 / engineer / mo. This is the route for teams looking for MLOps consulting that ends with the capability in-house.
• A new agent with the operating layer included. Gigabit Agents from $8,000 per agent, flat, with evals, guardrails and observability in the build fee, as our production AI agent stack describes.

Running cost is the line most AI budgets leave out. The hidden costs of AI projects puts a number on it.