Domain 4·16% of exam·30 concepts
Evaluation, Testing & Optimization
Define evaluation metrics and datasets, run A/B tests and iterative improvements, diagnose failure modes, and optimise token usage, latency, and cost with logging and observability.
Official prep for this domain
Anthropic's own free prep module for this part of the syllabus, on the official prep course. Free with an Anthropic Academy sign-in.
4.1
Define evaluation metrics (accuracy, latency, cost, safety, security)
4.2
Design evaluation datasets and test frameworks using mixed methodologies
4.3
Conduct A/B testing and iterative improvements
4.4
Diagnose system issues (prompt failure, hallucinations, model mismatch)
4.5
Optimize token usage, latency, and cost-performance trade-offs
4.6
Monitor system performance using logging and observability tools
- ccarp-4.6.1The Four Categories a Production System Must Log
- ccarp-4.6.2Request-Level Tracing vs Aggregate Metric Dashboards
- ccarp-4.6.3Anomaly Detection Thresholds and Alerting
- ccarp-4.6.4Discernment: Judging Output Quality, Not Just Watching Metrics
- ccarp-4.6.5Building the Technical-to-Business Metric Translation Layer
Other exam domains
Domain 1 · 17%Solution Design & ArchitectureDomain 2 · 13%Claude Models, Prompting & Context EngineeringDomain 3 · 19%IntegrationDomain 5 · 14%Governance, Safety & Risk ManagementDomain 6 · 14%Stakeholder Communication & Lifecycle ManagementDomain 7 · 7%Developer Productivity & Operational Enablement