LLMOps: A Complete Guide for Teams Running LLM Features in Production

What is LLMOps?

LLMOps is production operations for features built on large language models. This guide assumes that the feature is live or launching within a few months, and that it mostly calls a provider's API. Self-hosting models on GPUs and fine-tuning are separate problems, so they stay out of scope here.

MLOps still matters, but three things change. The prompt becomes a production artifact, the same input can return different outputs, and each call has a direct per-token cost. Teams that call an API often skip training pipelines altogether, and they still need everything in the table.

Why LLM features break differently

Nothing throws an error

A plausible wrong answer returns HTTP 200. Latency and error monitoring can't see it, so the first signal is often a customer or a support ticket.

Damage can also be narrow. At SREcon in 2025, practitioners described quality falling in one small question space while common questions stayed fine (TechTarget). An average score across all traffic can hide that.

The model underneath you changes

Providers publish retirement dates, and the dates arrive whether or not you're ready. These two notices were on the providers' pages when this guide was written:

Read the date, not just the policy. OpenAI's page states at least six months for generally available models, unless safety or compliance concerns require less, yet this notice ran about three months. Plan around the shortest window you've actually seen.

Many teams were exposed. In March 2026, GPT-4o had 22% adoption and Sonnet 4.5 had 19% among Datadog's LLM customers. One speaker at SREcon compared a model change to swapping the underlying database, which is a useful way to size the risk.

Cost and capacity move with usage

Average tokens per request more than doubled year over year for the median customer in Datadog's report, which covers telemetry from more than 1,000 of its own customers. Rate limits were 60% of LLM errors in February 2026 and almost a third in March. That's one vendor's customer base rather than the whole market, but it shows a failure mode many LLMOps guides skip.

Not every feature needs all of this on day one.

When do you need LLMOps, and how much?

You need some of it as soon as any of these is true:

• Real users act on the answers.
• The feature touches customer data.
• Your bill has moved and nobody can explain why.
• You've received a retirement notice, or your model has a published retirement date.
• More than one person changes prompts.

More teams are reaching this point. In McKinsey's 2026 State of AI survey, 44% of respondents said AI is scaling across their enterprise, up from 38%. That's self-reported and covers AI broadly, but the direction is clear.

Scale the setup to the team. This table is our guidance, not an industry standard:

Whatever the size of the team, the same loop applies.

How LLMOps works: the operating loop

Every change goes through the same cycle. You test it, gate it, roll it out in stages, and watch live traffic. Whatever goes wrong becomes a new test case.

A minimal LLMOps architecture has four parts: prompts and model configuration in version control, an evaluation runner in CI, a gateway or shared client on the request path, and a trace store that sampled scoring reads from. You can start with a small version of each.

A loop with no owners stops at the first incident.

Who owns what when an LLM feature is live

Ownership is the question most teams leave open. At SREcon in 2025, one engineer framed it as "Who can fix it?", and asked whether data scientists should be on call (TechTarget). SREs can shed load on an overloaded database. A hallucination needs someone who can change the prompt or the model.

An ownership map

Use this as a default and change the names to fit your team:

Who gets paged

Split failures by who can fix them.

Start with the one thing every later step relies on: a test set.

Step 1: Build a test set that catches regressions

A recurring question in practitioner threads is how to catch regressions after a prompt or model change. The answer starts here. Nothing else in LLMOps can tell you whether a change made answers worse.

What goes in it

Some teams call this a golden dataset. Each case has an input and a way to judge the output, which might be a reference answer, a list of required facts, or a rule. Cover these groups:

• Common requests: a sample of real production traffic, with personal data removed.
• Difficult cases: long inputs, mixed languages and messy formatting.
• Ambiguous requests: the feature should ask a question or state an assumption.
• No-answer cases: the right response is "I don't know", or a refusal.
• Adversarial cases: prompt injection, attempts to reveal data, and policy violations.
• Past failures: every real incident, which is often the most valuable group.

Tag each case by topic and customer segment, so that a narrow regression shows up in its own score. Start with 50 to 100 cases. A small set that runs on every change beats a large one that nobody maintains.

LLM evaluation metrics that matter

Match LLM evaluation metrics to what the feature does:

Several ranking guides still present BLEU and ROUGE as key metrics. They measure text overlap with a reference, so a correct answer in different words can score low, and a wrong answer with similar wording can score high. They don't tell you whether a support answer is right.

Always add two inexpensive checks: output length and cost per request. Both can shift when a model changes, even if quality holds.

LLM-as-a-judge and human review

Use the cheapest method that gives a reliable result:

• Deterministic checks for schemas, required fields, banned content and citations. Run them on every case.
• LLM-as-a-judge for correctness and groundedness at scale. A judge model is also a model, so pin its version and don't change it during a comparison.
• Human review to calibrate the judge and to settle high-stakes cases. A framework for model migration from Verint researchers calibrates automated metrics against human judgments, so that comparisons hold up with limited manual review.

Setting pass marks

Product and the AI engineer should set pass marks together, because the numbers express what the business will accept. Use two kinds:

• Must-pass cases: injection, data exposure and regulated advice. One failure blocks the release.
• Comparison limits: how far a metric can move from the current production baseline.

Absolute scores matter less than change against the baseline. Watch for noise too. In a set of 200 cases, one case is worth half a point, and outputs vary between runs, so repeat each case a few times before you trust a small move. This test set is the base of your LLM evaluation framework, and it gets its real value when CI runs it automatically.

Step 2: Version prompts and models, and gate every change

Prompt versioning: treat prompts as releases

A prompt edit can change behavior as much as a code change, so give it the same path to production:

1. Keep prompts in version control, or in a registry that keeps full history.
2. Point production at a named alias, never at "latest", so that promotion is deliberate.
3. Log the prompt version, the requested model and the responding model on every request.
4. Pin dated model snapshots where your provider offers them, so that the model doesn't change without a decision from you.

Don't assume a provider-hosted prompt store is permanent. OpenAI scheduled its reusable prompt objects and v1/prompts API to shut down on November 30, 2026, and tells users to move that content into application code.

A release gate in CI

Run the gate on any change to a prompt, model setting, retrieval configuration or tool definition. The file below shows the policy to encode. The field names are illustrative, and the numbers are starting points to tune:

# Illustrative release gate. Not a specific tool's schema.
feature: support-assistant
run_when_changed:
  - prompts/support/**
  - config/model.yaml
  - retrieval/**
compare_against: production
test_set: evals/support-golden-v14
repeats_per_case: 3

must_pass_tags: [injection, data_exposure, regulated_advice]

limits:
  correctness:        { minimum: 0.90, max_drop: 0.02 }
  groundedness:       { minimum: 0.92, max_drop: 0.02 }
  refusal_rate:       { max_change: 0.03 }
  p95_output_tokens:  { max_increase_pct: 20 }
  cost_per_request:   { max_increase_pct: 15 }
  any_tag_correctness: { max_drop: 0.05 }   # catches narrow regressions

on_failure: block_merge
override_requires: [feature_owner, product_owner]

The any_tag_correctness limit matters most for silent regressions. It stops a change that holds the overall average but damages one topic.

Roll out in stages

Passing the gate means a change is ready for real traffic, not that it's safe:

1. Shadow traffic: send a copy of live requests to the candidate and score the output without showing it to users. Don't execute tool calls that change state.
2. Canary: serve 5% of traffic, keeping each customer on the same version. Compare quality samples, refusals and cost.
3. Expand: move to 25%, then 100%, as long as each stage holds.
4. Rollback: make it a configuration change that takes seconds and needs no deployment.

Here's how those two steps hold up against a real deadline.

Worked example: surviving a model retirement

This scenario is illustrative. The feature and its numbers are invented to show the method, and they aren't a client result.

A support assistant answers billing questions from a help center. Its provider announces that the model will retire in 60 days.

The first comparison is where the value shows up:

On overall correctness alone, the replacement looks better. The team would have shipped a model that refuses twice as often, writes longer answers, and gets worse at refunds. So ask what changed for your users, along with whether the new model scores well.

Three lessons apply to any retirement:

• Measure more than correctness. Verint's production migration framework, demonstrated on a question-answering system with 5.3M monthly interactions, evaluated correctness, refusal behavior and stylistic adherence.
• Your rollback target changes. After the retirement date, the old model no longer exists. A rollback needs a second tested candidate, which is why the team keeps one ready.
• Automate the warning. OpenAI's List Models endpoint exposes an optional shutdown_date field. For other providers, check the deprecation page on a schedule. If you reach a model through a cloud platform, check that platform's own dates, as they can differ.

Tests catch what you predicted. Live traffic shows you the rest.

Step 3: LLM monitoring and observability

Traces show what happened, evals show whether it was right

LLM observability lets you reconstruct a single request: the prompt version, retrieved context, model, tokens, tools and output. LLM monitoring watches aggregate signals over time and alerts on them. Neither one tells you if an answer was correct. That takes evaluation on a sample of live traffic.

This differs from data observability, which watches pipelines and tables. You may need both, because a stale pipeline can feed bad context into a healthy-looking model.

How to monitor an LLM in production: the signals to watch

Alert on change from your own baseline. The conditions below are examples to tune:

Sampling live traffic

You can't score everything economically, so sample with intent:

• Random sample: a small share of all traffic, such as 1% to 5%, for an unbiased trend.
• Targeted sample: all requests with negative feedback, guardrail hits, or a human handoff.
• Per-topic minimum: a floor for each important topic or high-value customer, so that small segments can't hide in the average.

Send confirmed failures back into the test set each week. That's what keeps yesterday's incident from returning. Apply the same data handling to traces as to production data: redact personal data, set retention, and restrict access.

Trace anatomy and OpenTelemetry

Trace one request as nested spans:

support_answer                 customer=acme  feature=support  prompt=v37
├─ retrieve_documents          4 docs  index=help-2026-11  120 ms
├─ build_prompt                3,100 input tokens estimated
├─ chat <requested model>      response model recorded  1,920 ms
│    input 3,050  cached 2,400  output 310  estimated cost recorded
└─ output_guardrail            passed  18 ms

OpenTelemetry's GenAI semantic conventions give you shared names, including gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Add your own attributes for feature, customer and prompt version.

One caution: the conventions are still marked Development, and names have changed before. For example, prompt_tokens became input_tokens. Keep a mapping layer between your emitters and dashboards, so that a rename doesn't silently break cost reporting.

Step 4: Keep retrieval quality from drifting

If your feature uses RAG, then new embeddings, chunking rules, ranking settings or source documents are all releases. Run them through the same gate.

Test retrieval apart from the answer. If the right document never reached the model, prompt changes won't fix it. Track whether expected documents appear in the top results, and whether the answer stays grounded in what was retrieved. Also watch index freshness, because a broken sync can look exactly like a model regression.

If retrieval is your weak point, compare approaches in our guide to GraphRAG, vector RAG and SQL retrieval and our pgvector vs Pinecone comparison.

Step 5: LLM cost optimization and capacity

Calculate cost per feature and customer

Total spend doesn't tell you where to act. The method has three steps:

1. Tag at request time with the feature, customer, prompt version and model.
2. Compute from token logs, using a price table with effective dates, because prices change.
3. Reconcile to the invoice each month. If the numbers diverge, your tagging or price table is wrong.

An illustrative calculation, at $3 per million input tokens and $15 per million output tokens:

To estimate cost before shipping, run your test set, take the token counts, and multiply by expected volume. Add a margin for retries and for longer real-world conversations.

-- Illustrative. Assumes input_tokens excludes cached tokens.
SELECT
  r.feature,
  r.customer_id,
  SUM(r.input_tokens        * p.input_per_mtok  / 1e6
    + r.cached_input_tokens * p.cached_per_mtok / 1e6
    + r.output_tokens       * p.output_per_mtok / 1e6) AS cost_usd
FROM llm_requests r
JOIN model_prices p
  ON r.response_model = p.model
 AND r.created_at >= p.valid_from
 AND (p.valid_to IS NULL OR r.created_at < p.valid_to)
WHERE r.created_at >= DATE '2026-10-01'
  AND r.created_at <  DATE '2026-11-01'
GROUP BY r.feature, r.customer_id
ORDER BY cost_usd DESC;

Providers report cached tokens differently, so check whether cached tokens are already inside the input count. A mistake here quietly double-counts spend.

Levers that lower the bill

In order of effort:

1. Prompt caching. In Datadog's data, system prompts made up 69% of input tokens in March 2026, yet only 28% of LLM call spans on models that support caching showed cached-read tokens. Keep the stable content at the start of the prompt. Anthropic charges 0.1 times the base input rate for a cache read and 1.25 times for a five-minute cache write. In the example above, reading 1,500 of the 2,000 input tokens from cache would cut the request to about $0.0095, before write costs.
2. Shorter context. Remove tool definitions, documents and history that the request doesn't use.
3. Output limits. Set them per feature, and ask for concise formats where they fit.
4. Routing to smaller models. Send simple requests to a cheaper model. Each route is a release, so run it through the gate.

For a wider view of agent infrastructure spend, see the hidden infrastructure cost of enterprise AI agents.

Rate limits and spend caps

Providers limit requests and tokens per minute, and exceeding a limit returns HTTP 429. Not every 429 should be retried. On the Claude API, a normal rate limit includes a retry-after header. A monthly spend cap also returns a 429, but without retry-after, so every retry keeps failing until access resumes. Handle errors in this order:

1. Read the error code before you retry.
2. If the error is about spend or quota, stop retrying and alert a person.
3. If it's a rate limit, honor retry-after, then back off with jitter.
4. Queue work that can wait, and reduce traffic to features that can degrade.
5. Fail over only to a model that has passed your test set.

That last point gets missed. An untested fallback is an unreviewed release that goes live at the worst possible moment.

What an LLM gateway does

An LLM gateway is a shared layer between your application and providers. It can centralize keys, per-team limits, usage logs, retries and failover. It's useful once several features share providers. It isn't necessary for a single feature, where a thin shared client may be enough.

Step 6: Secure inputs, outputs and access

Prompt injection holds the top spot, as LLM01, in the OWASP LLM Top 10 2026. OWASP's earlier edition said it's unclear whether fool-proof prevention exists, so plan to limit the damage:

• Enforce permissions outside the model. Never rely on a system prompt as a security control, and don't put secrets in it.
• Make retrieval permission-aware. A user should only retrieve documents they're allowed to read.
• Treat outputs as untrusted. Validate and encode them before they reach a browser, database or another system.
• Use guardrails as one layer. In LLMs, guardrails are input and output checks for injection, personal data and policy. They reduce risk, but they don't remove it, and they add latency that you should measure.
• Keep an audit trail. Record prompt version, retrieved document IDs, tool calls and access decisions, and redact personal data.

Keep adversarial cases in your test set, so that a model change can't quietly weaken them. Legal and compliance sign-off stays with your own counsel. For tool access, see our guide to MCP server security, access control and guardrails.

Step 7: Respond to incidents when the answers are wrong

Practitioner write-ups disagree on severity levels and triage order (Tian Pan, Future AGI). Treat what follows as a starting point, and adapt it to your feature.

Decide severity by user harm

A service can be up and still be in a severe incident. Grade severity by what happens to users, not by system health:

Triage in order

Check containment first. Then check what changed, starting with what you control, because your own changes are quickest to confirm and already have a rollback.

A runbook you can copy

Every control named here has to exist before the incident. Test them during a calm week.

RUNBOOK: <feature> gives wrong or unsafe answers

Owner:            <team>
Quality on-call:  <rota>      Infra on-call: <rota>
Security contact: <name>      Support lead:  <name>
Dashboards:       <quality>   <traces>  <cost>
Provider status:  <url>

CONTROLS
Kill switch:      <flag>      Turns the feature off, shows <fallback message>
Prompt rollback:  <command>   Points the alias to the last good version
Model pin:        <config>    Tested fallback: <model ID>
Source disable:   <flag>      Removes a document source from retrieval

FIRST 15 MINUTES
1. Open an incident channel and assign one lead.
2. Collect 3 to 5 bad examples, with trace IDs.
3. Set severity by user harm. If data is exposed, use the kill switch.
4. Check recent changes: prompt, config, code, model, index, source data.
5. Roll back the most recent likely change. Don't wait for root cause.
6. Confirm recovery using the failing examples and a live sample.

NEXT
7. Tell support what users may have seen and what to say.
8. Preserve traces. Keep access restricted if they include personal data.
9. Find affected requests, so customers can be notified if needed.

AFTERWARDS
10. Add each failing example to the test set, tagged by topic.
11. Check why the gate and monitoring missed it, then change them.
12. Record the owner and date for each follow-up.

Turn every incident into a test case

The incident isn't closed until its failure exists in the test set. This step connects the end of the loop to the beginning, and it's why the test set grows more useful each quarter.

The steps stay the same, but the priorities change with the feature.

Real-world use cases: how the setup changes

These are typical patterns, not client stories:

When the feature becomes an agent

Everything above still applies once a model can take actions, but failures now change things in other systems. Add three controls: a separate identity for each agent, least-privilege permissions on each tool, and human approval for actions that are hard to undo. OWASP lists this risk as Excessive Agency. Before you connect agents to company data, read why your data stack may not be ready for AI agents.

LLMOps tools: what to buy, and how to stay portable

Buy by job, not by logo

Start with the job that's hurting most. Most teams need tracing first and evaluation next, then a gateway once several features share providers. Guardrail tools come in when your risk review calls for them. Buying a platform doesn't produce a test set or an owner, so do those regardless.

Tool comparison

This list covers tools that came up in our research rather than the full market. We don't sell or resell them. Prices and ownership were checked on November 4, 2026, and pricing units differ across vendors, so entry prices aren't directly comparable.

Stay portable

This market moves quickly. In 2026 alone, Langfuse joined ClickHouse and OpenAI announced its Promptfoo deal. OpenAI also scheduled its own Evals platform to shut down on November 30, 2026, pointing users to Promptfoo. These moves don't make any tool a bad choice, but your practices shouldn't depend on a single vendor's roadmap. Four habits help:

• Emit traces with OpenTelemetry where you can.
• Keep test sets and evaluation code in your own repo.
• Export a copy of your traces to storage you control.
• Know what license you'd fork from if you ever had to.

Tools don't prevent the most common mistakes.

Mistakes to avoid

1. Buying a platform before you have a test set. You get cleaner charts of a problem that you still can't measure.
2. Treating a prompt edit as a config change. It can alter every answer, so it needs a gate and a rollback.
3. Monitoring only latency and errors. Both stay healthy while answers get worse.
4. Trusting one average quality score. Narrow regressions disappear into it, so score by topic.
5. Building the test set once. It goes stale unless real failures keep feeding it.
6. Waiting for the retirement notice to test a replacement. Anthropic's minimum notice is 60 days, which leaves little room for a poor first result.
7. Keeping an untested fallback. During an outage, it becomes your least-tested release.
8. Tracking total spend only. You can't fix a cost that you can't assign to a feature or customer.
9. Retrying every 429. Some mean you've hit a spend cap, and retries only add noise.
10. Having no kill switch and no named owner. Both feel optional until the first bad afternoon.

Where to start

If you do one thing per week for a month:

• Week 1: Name an owner for quality, list every model ID in use, and check each one's retirement date.
• Week 2: Build 50 test cases from real traffic and past failures, then record your baseline.
• Week 3: Log prompt version, response model, feature and customer on every request, and add a kill switch.
• Week 4: Run the test set in CI on every prompt and model change, then copy the runbook and fill it in.

That won't cover everything in this guide, but it's enough to catch your next silent regression before your users do.

Build, buy or bring in a partner

You may not need a partner. If one feature is running and your team has time to work through the four weeks above, start there. If the feature matters to revenue and nobody owns quality, outside help can shorten the first pass.

DharmOps AIOps covers LLM features, agents and trained models that you already run in production or will launch within a few months.