How to Evaluate LLM Outputs: A Production Eval Guide

Knowing how to evaluate LLM outputs is the difference between shipping a Claude integration with confidence and deploying something you can only hope behaves correctly in production. This guide covers the full evaluation stack: grader selection, rubric design, golden dataset construction, and turning evals into a real release gate rather than a vanity dashboard.

Why do teams skip evals, and when is that actually defensible?

Teams skip evals for three reasons: they believe their task is too simple to warrant them, they lack the tooling, or they are optimising for shipping speed. The first reason is occasionally correct. A single-turn, deterministic transformation (convert this CSV to JSON) can be verified with a unit test. The second and third reasons are rationalisation.

The honest framing is this: every Claude integration already has an eval. It is called your users. The question is whether you run that eval in a controlled environment before release or in production after something breaks. For any integration that uses tools, maintains context across turns, or routes between subagents, informal evaluation is not faster; it is just deferred cost with compounding interest.

The CCAR-F exam reflects this directly. Domain 5, Context Management & Reliability, carries 15% of exam weight and tests whether candidates can design systems that remain reliable across extended sessions, not just ones that succeed on the first try.

What are the three grader types, and how do you choose between them?

Answer-first: choose your grader based on whether the correct output can be specified in advance. The three types are deterministic graders, LLM-as-judge graders, and human review, and they are not mutually exclusive.

A layered approach works best in practice. Run deterministic checks first because they are cheap and fast. Escalate failures and borderline passes to an LLM judge. Reserve human review for calibrating the judge and for any case where the judge returns low confidence.

For Claude agent pipelines specifically, the Agentic Loop Anti-Patterns concept is worth reviewing before you design your graders. Many eval failures trace not to bad model outputs but to structural problems in the loop itself: premature termination, stale context, or tool misrouting that the model cannot recover from.

How do you design evals that reflect real-world tasks rather than synthetic benchmarks?

Synthetic benchmarks fail because they optimise for the distribution of your test set, not the distribution of your users. Real-world evals start from real-world failures.

The practical method:

1. Instrument your integration to log every tool call, every model response, and every user action that follows.
2. Tag sessions where users retry, escalate, or abandon. These are your failure candidates.
3. For each failure candidate, reconstruct the full conversation including tool results. This is your replayable failure case.
4. Anonymise and add to your golden dataset with a human-assigned label.

The replay step is critical. A failure case you cannot replay is a failure case you cannot fix reliably. When a Claude Code or MCP run produces an unexpected result, capture the full context immediately: the system prompt, the conversation history, the tool definitions, and the tool results. Context degrades fast once a session ends.

{
  "case_id": "fc-2026-07-11-001",
  "domain": "tool_routing",
  "system_prompt_hash": "sha256:a3f...",
  "conversation": [...],
  "tool_definitions": [...],
  "tool_results": [...],
  "expected_behaviour": "Route to search_inventory, not search_orders",
  "observed_behaviour": "Routed to search_orders; returned empty result",
  "human_label": "fail",
  "severity": "high"
}

Storing system_prompt_hash rather than the full prompt keeps the dataset portable while still letting you detect when a prompt change invalidates old cases.

How do you build a layered rubric that separates distinct quality dimensions?

A single pass/fail score collapses information you need. A layered rubric scores each quality dimension independently so you can see exactly where a response degrades.

For a Claude agent that retrieves information and synthesises an answer, a four-dimension rubric looks like this:

Reproducibility deserves special attention for agentic systems. A response that is correct 7 times out of 10 is not a reliable response; it is a probabilistic one. The CCAR-F exam consistently rewards deterministic solutions over probabilistic ones when stakes are high. If your rubric does not measure stability across runs, you are measuring one-off success, not production reliability.

The Prompt Engineering & Structured Output domain covers how output schema design affects groundedness scores. A response that is forced into a structured schema is easier to grade deterministically and harder for the model to hallucinate into.

What belongs in a golden dataset for tool-using Claude integrations?

A golden dataset for a tool-using integration must cover four categories: happy paths, edge cases, adversarial inputs, and known failure regressions.

The regression category is the one teams most often neglect. Every time you fix a production failure, the fixed case should enter the golden dataset immediately. This is the mechanism that converts your incident history into a release gate.

Golden datasets for integrations that use external tools require additional maintenance discipline. Tool schemas change. MCP server responses evolve. A golden case that was valid in March 2026 may be invalid by July 2026 if the upstream tool changed its response format. Version your tool definitions alongside your cases.

The Tool Design & MCP Integration domain covers how tool description quality affects routing decisions. If your golden dataset includes tool misrouting failures, check whether the root cause is in the tool description before attributing it to the model.

How do you turn evals into a release gate rather than a dashboard metric?

A release gate has three properties a dashboard metric does not: it is automated, it is blocking, and it has a defined threshold. If your eval suite can produce a red number on a screen but cannot stop a deployment, it is not a gate.

The implementation pattern for a CI-integrated eval gate:

# Run eval suite against staging deployment
python run_evals.py \
  --dataset golden_dataset_v4.jsonl \
  --model claude-sonnet-4-5 \
  --grader-config graders/production.yaml \
  --output results/eval_run_$(date +%Y%m%d).json

# Check pass rate against threshold
python check_gate.py \
  --results results/eval_run_$(date +%Y%m%d).json \
  --min-pass-rate 0.92 \
  --required-dimensions "groundedness,reproducibility" \
  --fail-on-regression

The --fail-on-regression flag is the critical one. A deployment that improves average pass rate but reintroduces a known failure should be blocked. Aggregate metrics can mask regressions in specific cases; per-case regression checks cannot.

def check_regressions(current_results: dict, baseline_results: dict) -> list[str]:
    """Return case IDs that passed in baseline but fail now."""
    regressions = []
    for case_id, baseline in baseline_results.items():
        if baseline["label"] == "pass":
            current = current_results.get(case_id, {})
            if current.get("label") == "fail":
                regressions.append(case_id)
    return regressions

For multi-agent systems, the gate should also check dimension-level scores, not just overall pass rate. A system where groundedness drops from 0.95 to 0.78 while overall pass rate holds at 0.91 has a real problem that aggregate scoring hides.

How do you measure stability versus one-off success for multi-step agent runs?

Stability measurement requires running the same input multiple times and comparing outputs. For deterministic tasks, exact match across runs is the standard. For generative tasks, you need a similarity metric.

A practical stability protocol for a Claude agent:

1. Select 20% of your golden dataset cases at random for stability testing.
2. Run each selected case five times with temperature 1 (the default; do not artificially lower temperature to fake stability).
3. For structured outputs, compute exact match rate across the five runs.
4. For free-text outputs, compute pairwise semantic similarity using an embedding model and flag cases where minimum similarity falls below 0.85.
5. Report stability as a separate dimension in your eval results.

Cases that pass on average but show high variance are candidates for Prompt-Based vs Programmatic Enforcement. If a response is inconsistent, the fix is usually structural (constrain the output schema, add a validation step, use a hook) rather than prompt-level.

The Coordinator Responsibilities concept is relevant here for multi-agent systems. A coordinator that does not validate subagent outputs before synthesis will propagate instability upward. Eval design should test the coordinator's error handling, not just the subagents' outputs.

What is the right cadence for running evals, and how do you keep the dataset fresh?

Run your full eval suite on every pull request that touches the system prompt, tool definitions, model version, or orchestration logic. Run a lightweight smoke suite (top 20 cases by severity) on every deployment. Run stability tests weekly on production traffic samples.

Dataset freshness requires a maintenance schedule:

• Review and retire cases that have been passing for six consecutive months with no variance. They are no longer informative.
• Add at least one new case per production incident.
• Re-label cases when the expected behaviour changes due to a product decision, not a model failure.
• Version the dataset with a changelog so you can trace why pass rates changed between releases.

The Context Management & Reliability domain on the CCAR-F exam tests exactly this kind of systems thinking: not just whether you can build an eval, but whether you can design one that remains valid as the system evolves. Our concept library at /concepts covers 174 atomic concepts mapped to all five exam domains, including the reliability patterns that underpin production eval design.