LLM Evals Guide: Production Agent Reliability

This llm evals guide is written for teams shipping Claude agents and MCP integrations into production, not for researchers chasing leaderboard positions. We cover how to build an eval suite from real failures, how to balance automated grading with human review, and how to use evals as a release gate rather than a postmortem report. Every recommendation here maps to the practical judgment that the CCAR-F architect exam rewards.

Why do benchmark-style evals fail in production?

Benchmark evals fail because they test the distribution of prompts the benchmark author imagined, not the distribution your users actually send. When you evaluate a Claude agent against a curated academic set, you are measuring performance on a population that may share almost no statistical overlap with your production traffic.

The failure mode is predictable: scores look strong in staging, then the agent misbehaves on the first real edge case a user hits. The root cause is almost always a mismatch between the eval dataset and the true task distribution.

The fix is to seed your eval suite from production logs, not from imagination. Collect the inputs that caused failures, near-misses, or escalations in the last 30 days. Annotate them with the correct output or the correct decision. That corpus is your golden dataset, and it is worth more than 500 synthetically generated prompts.

How do we build evals from real production failures?

Build from failures in three steps: capture, triage, and formalise.

1. Capture. Log every agent turn with a unique trace ID. Include the full input, the tool calls made (with arguments), the tool results returned, and the final output. For agentic loop workflows, log the stop_reason at each turn so you can distinguish a clean finish from a premature termination.

2. Triage. Review failures weekly. Classify each one into a small taxonomy: wrong tool selected, correct tool but wrong arguments, correct output but unsafe side-effect, refusal when action was appropriate, action when refusal was appropriate, or output format violation. This taxonomy becomes your eval category labels.

3. Formalise. For each failure, write a test case: the input, the expected behaviour (which may be a range of acceptable outputs rather than a single string), and the grading criterion. Keep the case as narrow as possible. A test case that covers two failure modes at once is harder to maintain and harder to interpret when it fails.

A minimal test case record looks like this:

{
  "id": "prod-failure-2026-06-14-007",
  "category": "wrong_tool_selected",
  "input": {
    "system": "You are a data pipeline assistant.",
    "user": "List all CSV files modified in the last 24 hours under /data/ingest."
  },
  "expected_tool": "glob",
  "forbidden_tools": ["grep"],
  "grading": "exact_tool_match"
}

Aim for 20 to 50 high-signal cases before you invest in scale. A suite of 30 cases that each trace to a real production failure will catch more regressions than 300 synthetically generated cases that do not.

What should evals measure for Claude agents and MCP workflows?

Four dimensions matter for production reliability:

Tool-use safety deserves special attention in MCP integration workflows. An agent that calls a write tool when a read tool was sufficient, or that passes unsanitised user input as a tool argument, represents a safety failure even if the final output looks correct. Your evals must instrument tool calls, not just final responses.

For context management scenarios, add a fifth dimension: context fidelity. Did the agent preserve attribution and avoid hallucinating facts from earlier in a long session? This is particularly relevant for workflows that exceed 50,000 tokens of context, where the attention dilution effect becomes measurable.

How do we define success criteria that are specific and measurable?

Vague criteria produce unreliable evals. "The response should be helpful" is not a criterion; it is a wish. Measurable criteria have a binary or ordinal outcome that two independent reviewers would agree on.

Use this template for each eval category:

CRITERION: [category name]
PASS condition: [exact, observable property of the output or tool call]
FAIL condition: [exact, observable property that constitutes failure]
AMBIGUOUS: [edge cases that require human review]
GRADING: [deterministic | llm-judge | human]

For example, a criterion for a structured output task in prompt engineering workflows:

CRITERION: schema_compliance
PASS: output is valid JSON that satisfies the declared JSON Schema with no additional keys
FAIL: output contains keys not in the schema, or is not parseable JSON
AMBIGUOUS: output is valid JSON but a required field contains an empty string
GRADING: deterministic (JSON Schema validator)

Writing criteria this way forces you to decide upfront what "good" means. It also makes it possible to automate grading for the deterministic cases and route only the genuinely ambiguous ones to human review.

What is the right balance between automated grading and human review?

Automated grading scales; human review calibrates. You need both, but in different proportions depending on the eval category.

LLM-as-judge is a practical middle ground for quality dimensions that resist deterministic grading. The pattern is to send the agent's output to a separate Claude call with an explicit rubric and ask for a structured verdict. The key discipline is to write the rubric before you see any outputs, not after, to avoid anchoring the rubric to the outputs you happen to have.

import anthropic

client = anthropic.Anthropic()

def llm_judge(task_input: str, agent_output: str, rubric: str) -> dict:
    prompt = f"""You are an impartial evaluator. Score the agent output below.

TASK INPUT:
{task_input}

AGENT OUTPUT:
{agent_output}

RUBRIC:
{rubric}

Respond with JSON: {{"score": 0|1|2, "reason": "one sentence"}}"""

    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=256,
        messages=[{"role": "user", "content": prompt}]
    )
    import json
    return json.loads(response.content[0].text)

Human review should be reserved for cases where the automated grade is uncertain, where the failure category is new, and for a random 5 to 10 percent sample of automated passes to catch systematic grader drift.

How do we prevent evals from being gamed?

An agent that recognises test conditions and behaves differently in evaluation than in production is a real risk, particularly as models become more capable. Three practices reduce this risk.

First, do not use a fixed eval prompt header that signals "this is a test." Vary the framing. If your production system prompt says "You are a data pipeline assistant," your eval system prompt should say exactly the same thing, not "You are being evaluated."

Second, draw eval inputs from production logs rather than constructing them synthetically. Production inputs carry the natural variation and noise of real user traffic. Synthetic inputs tend to be cleaner and more canonical than real ones, which makes them easier for a model to pattern-match as test cases.

Third, hold a blind test set. Maintain a partition of your golden dataset that is never used for iterative development or prompt tuning. Run it only at release gates. If your development set score improves but your blind set score does not, you have overfit to the development distribution.

How should evals function as a release gate?

Evals that only run as postmortems are retrospective. Evals that run as release gates are preventive. The difference is when you run them and what authority they have to block a deployment.

A practical release gate pipeline for a Claude agent or Claude Code workflow looks like this:

The key design decisions are:

• Safety assertions are hard gates. A single failure blocks the merge.
• Quality scores are soft gates. A drop below baseline triggers human review, not an automatic block, because quality regressions sometimes reflect intentional trade-offs.
• The blind test set runs only at merge time, not during development, to preserve its integrity as an unbiased signal.

Define your regression threshold before you ship. A common starting point is: no more than two new failures on the golden dataset per release, and zero new failures on the safety assertion suite.

How do we keep a golden dataset current?

A golden dataset that is not maintained becomes stale. Agent behaviour changes, user traffic shifts, and the failure modes you captured six months ago may no longer represent the failures you will see next month.

Maintain the golden dataset with a quarterly review cycle at minimum:

1. Add. After each production incident or escalation, add a new test case within one sprint. This keeps the dataset anchored to current failure modes.
2. Retire. Remove cases that no longer represent plausible inputs, for example cases tied to a feature that has been deprecated.
3. Regrade. When you change a grading criterion, regrade all existing cases under the new criterion before comparing scores across releases.
4. Version. Tag the dataset with a version number and record which model and configuration produced the baseline scores. Score comparisons are only meaningful within the same dataset version.

For long-running workflows with heavy context, add a dedicated section of the golden dataset that exercises context management edge cases: sessions that approach the context window limit, sessions where earlier facts are contradicted later, and sessions where the agent must synthesise across many tool results. These cases degrade silently in production and are rarely covered by standard eval suites.

How do evals map to the CCAR-F exam domains?

If you are preparing for the Claude Certified Architect, Foundations exam (CCAR-F, $125, passing score 720 on a 1,000-point scale), eval design appears most directly in Domain 5 (Context Management and Reliability, 15%) and Domain 1 (Agentic Architecture and Orchestration, 27%). The exam consistently rewards deterministic solutions over probabilistic ones when stakes are high, which maps directly to the hard-gate / soft-gate distinction above.

The table below shows which eval concepts align to which exam domains:

Our concept library covers 174 atomic concepts mapped to all five domains and 30 task statements, including the reliability and evaluation patterns that appear in scenario-based exam items.