Selecting the Correct Fix for Model Mismatch

The failure whose fix is a different model, verified

Prompt failure and hallucination both live inside a fixed model; model mismatch is the class where the model itself is the wrong choice. This apply-level knowledge point is about recognizing model mismatch as its own class and applying the fix it demands, eval-gated model selection, rather than defaulting to a prompt tweak. It also closes the loop on the taxonomy: with prompt failure, hallucination, and model mismatch each mapped to a distinct fix category, you have a clean decision structure for single-call failures.

What model mismatch is

Model mismatch happens in two related ways. The first is that the model tier chosen is simply wrong for the task's complexity, a tier too weak to handle the reasoning the task requires, so it fails on the hard cases no matter how the prompt is written. The second is that a model was swapped, for cost, for availability, for a version update, without re-evaluating whether the new model still meets the task's requirements. In both cases the root cause is the model selection, not the instruction and not missing grounding.

Recognizing this class matters because its symptoms can resemble the others, quality is poor, answers are wrong, but the lever that fixes it is different. No amount of prompt clarification or grounding compensates for a model tier that cannot do the task.

The fix is eval-gated model selection

The fix for model mismatch is to select the model gated behind an eval run. That means choosing the tier appropriate to the task's complexity and then confirming that choice against the eval suite, letting the evals, not intuition, decide whether the model fits. The eval gate is what distinguishes this fix from a guess: you do not simply believe the new tier is better, you measure it against the acceptance criteria before trusting it in production.

This is why the fix is a model change and not a prompt edit. A prompt edit operates within the current model; a model mismatch requires stepping up (or down) to a tier that matches the task, and the only reliable way to know the new choice actually resolves the problem is to run the suite that defined success in the first place.

Never swap a model without re-running evals

The sharpest operational rule here is that swapping a model without re-running the eval suite is dangerous, because a swap can silently reintroduce a previously-fixed failure class. A model that was fine on cost or latency might, on the specific behaviours your evals cover, fail in a way the old model did not, perhaps reintroducing a hallucination pattern you had grounded away, or an edge-case error you had tuned out. Without re-running the evals, that regression ships invisibly, because, as always, the LLM system does not crash, it just gets quietly worse.

So the eval suite is the gate on both directions of the fix: you use it to confirm the new tier resolves the mismatch, and you use it to confirm the swap did not break anything else. This is the evals-as-a-gate discipline applied specifically to model changes.

What the exam trips candidates on

The first trap is fixing an apparent model-mismatch issue by rewriting the prompt instead of re-evaluating whether the model tier fits the task. A scenario will show a task the current tier cannot handle and a team endlessly tuning the prompt; the credited reading identifies the mismatch and prescribes an eval-gated model change, noting that prompt edits cannot compensate for an inadequate tier.

The second trap is swapping to a different model tier to resolve a quality issue without re-running the eval suite to confirm the swap actually fixed it. A scenario will show a team that changed models and assumed the problem was solved; the correct answer insists on re-running the evals, both to confirm the fix and to catch any failure class the swap silently reintroduced.

Common misreadings to avoid

How this shows up on the exam

Domain 4 questions on this knowledge point present a quality problem that persists through prompt edits, or a model swap trusted without re-evaluation. The reliable moves are to recognize model mismatch as its own class, fix it with eval-gated model selection rather than a prompt tweak, and never treat a model swap as done until the eval suite confirms it.

This completes the single-call classes from the core failure taxonomy, sitting alongside prompt failure vs hallucination so each maps to a distinct fix. The eval gate is evals as acceptance criteria applied to model changes, the validation-before-exposure logic connects to live vs shadow testing, and an unannounced model swap is one of the causes disentangled in change attribution.