The Six-Step Loop for Testing LLMs in Production
LLMs behave differently in production than they do in test environments. This six-step loop closes the gap between pre-production evaluation and real-world performance.
You have probably lived this. A CI fixture confirmed the agent called the right tool. A judge said the response was grounded. Everything was green. Then a real user asked something slightly off-distribution, the retriever missed, the model produced a confident wrong answer, and the support queue lit up. Nothing was broken in the test suite. The failure lived in the gap between what you tested and what production actually sends.
Closing that gap is not a single test. It is a loop that runs continuously: instrument, score, gate, simulate, sample, optimize. The tools underneath change from team to team, but the loop does not, and the honest version of it does not depend on any one vendor. What follows is the shape of the loop and why each step earns its place, and the longer engineering write-up walks each step through its code and thresholds.
Why LLM testing breaks in 2026
Three things shifted in the last year, and together they broke the old pre-deployment-only approach.
Public benchmarks stopped being production proxies. As the well-known benchmarks aged, later model releases had almost certainly seen them during training, so a high score measures memorization more than capability. The contamination-resistant versions, drawn from actively maintained repositories, show frontier models scoring materially lower, often twenty to thirty points below their headline numbers. The lesson is not the exact figure, it is that any benchmark the labs have trained on is no longer a capacity plan. Private fixtures and sampled production traces are the only credible gates left.
The agent surface area exploded. The current stack is multi-step, multi-tool, multi-model, and often multi-modal. A single support agent runs retrieval, tool calls, parallel branches, dynamic routing, and model fallbacks. Scoring only the final answer misses every interesting failure in that stack: silent retries, retrieval drift, wrong tool selection, runaway loops, and injection through retrieved content.
Security stopped being a separate audit. The OWASP LLM risk list added system-prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption to the existing injection and disclosure categories. Testing for those now belongs in the gate, not in a quarterly review.
The combined effect is that pre-deployment unit tests no longer fit the shape of the failures that show up after release. You need a loop that connects live traces, evaluations, datasets, simulation, and prompt work.
The six-step loop at a glance
If you take one thing from the table: the gap between CI and production closes when failing live traces become both regression fixtures and optimization inputs in the same loop.
Step 1: Instrument every call
Every call you do not capture is a failure you cannot debug. The standard now is to emit spans using the shared tracing conventions, so each model call, tool call, and sub-agent call shows up as its own span with the model name, token counts, finish reason, arguments, and return values.
After instrumenting, verify four things: the root span shows the user request and final response, each model call is a child span with its details, each tool call is a separate child span, and sub-agent calls nest rather than flatten. If every step sits flat under the root with no nesting, context propagation is missing, and it has to be fixed before scoring, because scores attach to spans and wrong spans mean wrong scores.
Step 2: Score the spans
The pattern that mattered a year ago, running evaluators in a separate notebook and stitching scores back by ID, is gone. Scores now attach to the active span as it runs.
The workhorse is two evaluators together. A model-as-judge handles the semantic claims: groundedness, factual accuracy, instruction adherence, conversation coherence. A heuristic handles shape: schema validation, length, refusal rate, tool-argument validity, and simple contains or excludes checks. The judge is expensive and approximate, the heuristic is cheap and exact, and each misses things the other catches.
Two disciplines keep this honest. Calibrate the judge against a human-labeled set monthly and only trust it on routes where it agrees with humans past about eighty-five percent, treating the rest as advisory. And watch the cost, because a judge on every span can cost several times the inference bill. The fix is to route by cost class, which is the next step’s sampling math.
Step 3: Gate releases in CI
CI is the cheapest place to catch a regression. The minimum fixture set is five files: canonical success cases, known tricky edge cases, a regressions file, a safety file for injection and jailbreaks, and a thresholds config.
The rule that compounds is that every previously fixed bug becomes a fixture. After six months, that regression set is the most valuable thing in the test directory, because it encodes everything the team learned the hard way. Keep the thresholds tight, exact or in-order matching on tool trajectories and a high bar on response match, because a loose comparator passes an agent that called the right tools in the wrong order or produced word overlap without meaning. If you cannot hit the bar, the agent is not ready or the comparator is wrong, and lowering the threshold is never the fix.
Two anti-patterns to avoid: gating on pass or fail without recording the score, which loses the drift signal, and gating on a single judge with no heuristic beside it, which lets a judge regression wave through an obviously broken output.
Step 4: Simulate multi-turn scenarios
Single-turn fixtures catch a lot and miss cross-turn behavior, where the agent looks fine on each turn and fails the conversation as a whole. Simulation drives a persona through the agent for several turns, captures the transcript, and scores it with conversation-level rubrics for goal completion, trajectory quality, and tool use.
Pick personas that map to real failure classes, not synthetic edge cases: the user who interrupts mid-tool-call, the one who supplies contradictory information across turns, the one attempting injection through retrieved content, the one who never gets to the point, and the one with adversarial context in their profile. Five personas cover most of the long tail, and you add one for every production incident that traced back to a multi-turn behavior. The persona library is the simulation version of the regression set.
Step 5: Sample production traffic
Production is where the distribution actually lives. CI fixtures are a sample of what you thought would happen, and live traffic is a sample of what is happening. The gap between the two is your distribution shift, and the only way to measure it is to score real traces continuously.
The sampling shape that works has four tiers:
Run heuristics inline because they are cheap and catch shape errors that should never reach a user. Run judges on a background worker so eval latency never blocks the response. And stratify, because uniform sampling mostly returns happy-path noise. Stratify by route, user segment, outcome, and risk class, and sample everything where a guardrail tripped or a user left negative feedback. An agent doing a million requests a day sampled at one percent gives ten thousand mostly-uninteresting spans, while stratifying by escalated outcomes gives a few hundred that are actually informative. The output of this step is a live, growing dataset of failing traces, which feeds the last step.
Step 6: Optimize prompts on failures
The failing-trace dataset turns prompt iteration from guesswork into search. You pick a search method, pick an evaluator that grades the failure mode, and let it search prompt variants against the real failures. The output is a variant with a measurable lift on that dataset, which you then promote through the same CI gate from step three. If the variant fails the gate, you learned that the optimizer over-fit the failures, so you widen the dataset and run again.
That is where the loop closes. Optimized prompts ship through the gate, get instrumented in production, get scored on live traces, surface new failures, and feed back into the dataset.
A failure the loop catches
Consider a checkout agent on a four-step path: retrieve product info, validate cart, check inventory, confirm purchase. Fixtures cover the happy path, and production is fine for two weeks. Then the tool-trajectory score ticks down four hundredths over four days. No incident, no alert at the old threshold.
Step five is sampling high-value carts with a frontier judge, and its grounding score dips on the inventory step. The trace tree shows why: a vendor update added a “temporarily unavailable” status the agent had never seen, and because the prompt only listed in-stock, out-of-stock, and backorder, the agent read the new value as available. Three actions follow. The new case goes into the regressions fixture. The failing traces feed the optimizer with a rubric for status handling, which surfaces a prompt that handles all the values explicitly. And the variant ships through the gate behind a flag with the new prompt scored against the old.
Without the sampling, that sits in production until a customer calls. Without the optimizer, the fix is whatever the on-call wrote at 11pm. Without the regression fixture, the next vendor update reintroduces it. The loop is the only structure that connects all three.
Hardening it for real traffic
The six steps are the protocol. A few things harden it for production.
Cost. Eval can outrun inference on a naive setup, since a frontier-judge call on a large rubric can run several cents each. The four-tier sampling table is the control: heuristics on everything, cheap judges on a small slice, frontier judges on the risky fraction. Caching identical judge calls, using small judge models for narrow rubrics, and batching cut the bill further.
Latency. Anything that calls a model goes async on a worker queue that writes scores back as span attributes. If a judge must run inline as a guardrail, pin it to a small model and a strict timeout, and fall back to the heuristic on timeout.
Drift. Ship at least four monitors: score drift per route against a rolling baseline, tail latency, cost per request, and refusal rate. Alert on relative change, not absolute thresholds, because slow drift never trips a fixed line.
Reproducibility. If you cannot replay a failure you cannot fix it, so capture the full prompt, tool inputs and outputs, model version, retrieval results, and any seed. Scores without enough context to replay the span are a leaderboard, not a debugging tool.
The same six mistakes show up on most setups. Grading only the final response on a multi-step agent, so a lucky correction hides an earlier hallucination. Trusting a judge as ground truth without calibrating it against human labels. Leaning on public benchmarks as a shipping bar instead of private fixtures. Sampling production uniformly instead of stratifying by risk. Writing fixtures loose enough to pass a wrong agent. And ignoring cost and latency in the gate, so a prompt that lifts quality but doubles tokens counts as a win. The fix in every case is the same: make the gate fail on the real failure mode, not on a proxy for it.
The loop is what turns all of this from a pile of one-off tests into a system that gets sharper every week, because each production failure becomes a fixture the next release has to clear.