Domain 4·16% of exam·30 concepts
Evaluation, Testing & Optimization
Define evaluation metrics and datasets, run A/B tests and iterative improvements, diagnose failure modes, and optimise token usage, latency, and cost with logging and observability.
4.1
Define evaluation metrics (accuracy, latency, cost, safety, security)
4.2
Design evaluation datasets and test frameworks using mixed methodologies
4.3
Conduct A/B testing and iterative improvements
4.4
Diagnose system issues (prompt failure, hallucinations, model mismatch)
4.5
Optimize token usage, latency, and cost-performance trade-offs
4.6
Monitor system performance using logging and observability tools
- ccarp-4.6.1The Four Categories a Production System Must Log
- ccarp-4.6.2Request-Level Tracing vs Aggregate Metric Dashboards
- ccarp-4.6.3Anomaly Detection Thresholds and Alerting
- ccarp-4.6.4Discernment: Judging Output Quality, Not Just Watching Metrics
- ccarp-4.6.5Building the Technical-to-Business Metric Translation Layer
Other exam domains
Domain 1 · 17%Solution Design & ArchitectureDomain 2 · 13%Claude Models, Prompting & Context EngineeringDomain 3 · 19%IntegrationDomain 5 · 14%Governance, Safety & Risk ManagementDomain 6 · 14%Stakeholder Communication & Lifecycle ManagementDomain 7 · 7%Developer Productivity & Operational Enablement