Score accepted results, not just model preferences. Define task-specific checks for correctness, required fields, citations, tool arguments, policy compliance, and user-visible completion. Combine automated validation with human review for cases where a parser cannot judge usefulness. Track pass rate, critical failure rate, uncertainty, and the examples that changed from good to bad. A release that wins an average score but creates a new severe failure mode may be a regression. Measure reliability and latency separately from quality. Compare time to first token, complete response time, P95 and P99 latency, timeout rate, provider errors, retries, fallback share, and cancellation success.

Trace which contract was assumed and which one actually broke. Compare cohorts, not just totals. Keep a stable control route or baseline fixture when possible. Slice results by model, provider, region, tenant, prompt length, output length, workload class, language, streaming mode, and client version. A global average can hide a severe regression for one customer or a small but important structured output workflow. Use confidence and sample size appropriate to the decision. Do not call a noisy canary a success because its first few requests looked fine. Investigate observability gaps as part of the failure. If you cannot tell which prompt, route, capability policy, or fallback produced a result, the missing context is itself a reliability defect.

Compare schema pass rate, tool call accuracy, refusal patterns, latency, token use, and cost per successful task. During a canary, cap traffic and define rollback thresholds before the experiment starts. A gateway such as CrazyRouter can centralize the route split, but the acceptance criteria must belong to the workload owner. Watch for silent semantic drift. A model may keep the same schema while changing how it interprets a field, when it calls a tool, or how often it refuses. Add assertions about invariants and important decisions, not only snapshots of exact wording. Prefer rubric-based evaluation, deterministic checks, and human review for a small sample over brittle string comparisons.