Evals in Practice

I The Eval Pipeline

In Part 1, we explored what “good” means for AI. But knowing how to judge outputs is half the battle. The other half is building the system that makes evaluation repeatable and part of your shipping workflow.

A real eval pipeline has five stages: test set (curated inputs + expected behaviors), model run (generate outputs), scoring (automated checks + human review), analysis (patterns in failures, not just individual misses), and decision (ship, iterate, or hold).

II Building Test Sets

Your eval is only as good as your test set. And building a good one is harder than it sounds — you’re testing for the full range of human weirdness your AI will encounter in the wild.

A strong test set needs four categories:

The 80/20 rule applies in reverse: 20% of test cases catch 80% of issues — but you won’t know which20% until you’ve built all of them. The edge case you dismissed as irrelevant is the one that goes viral on Twitter.

III The Non-Determinism Problem

Here’s what makes AI testing fundamentally different: you can run the same test twice and get different results. This isn’t a bug — it’s a feature. Your model is, by design, a slightly different employee every time you ask it something.

LLMs have a temperature parameter that controls randomness. At temperature 0, outputs are nearly deterministic. At temperature 1, they vary wildly in style, structure, and accuracy.

This means a single test run tells you almost nothing. The model passed the safety check this time? Would it pass the next 9 times? Serious eval pipelines run each test case 5-10 times and evaluate the distributionof results. 9/10 passes = 90% reliability, probably fine. 7/10 = 70%, probably not. And 70% isn’t a hypothetical worst case — CMU’s TheAgentCompany benchmark found failure rates up to 70% on realistic workplace tasks. Meanwhile, METR research shows agent performance roughly doubling every seven months, and Stanford HAI’s data confirms it: SWE-bench coding scores leapt from around 60% to roughly 80% in a single year, while general computer tasks (OSWorld) went from 12% to 66%. The capability is accelerating fast, but the frontier is jagged — agents still fail one in three attempts on structured benchmarks.

IV Regression Testing & Ship Decisions

Regression testing for AI means running your full eval suite before and after every change and comparing the results. Did accuracy drop? Did safety scores change? Did latency increase? Did new failure modes appear? This is the beating heart of the eval pipeline — the thing that tells you whether a change made things better, worse, or just different.

The tricky part is that AI metrics are inherently noisy. A 2% accuracy drop might be random variance from non-determinism, or it might signal a real regression caused by your prompt change. This is why you need clear thresholds set before you start — not after you see the results. “We ship if accuracy is at least 92% AND safety score is at least 98% AND p95 latency is under 3 seconds.” Write these down. Share them with the team. Don’t negotiate with yourself after the numbers come in.

Sometimes you ship with known gaps, and that’s okay. A model that’s 95% accurate today beats one that’s 99% accurate next quarter — if the failure cases are low-severity and you have good fallbacks. The key is making that tradeoff deliberately, not accidentally. Stanford HAI’s 2026 AI Index found that improving one responsible AI dimension (say, safety) can measurably degrade another (say, accuracy) — so the tradeoff isn’t theoretical. It’s baked into the models themselves. Ship fast, but ship with eyes open — not just hoping nobody screenshots the bad outputs before you push a fix.

And when the numbers are ambiguous — accuracy up slightly, safety down slightly, a new edge case that didn’t exist before — that’s where your judgment earns its keep. No formula will tell you the right answer. But having the data in front of you makes the judgment call immeasurably better than guessing.

V Wrapping Up

Four things to take from these first two parts — before Part 3 goes deeper into the harness itself:

1. Evaluation is a product problem.You own what “good” looks like. The bar is yours to set — whether you measure it yourself or work with your team to build the harness.
2. Start with vibes, build toward pipelines. Manual review gets you started. Automated evals get you to scale and ship with confidence.
3. Think in distributions, not individual outputs. AI is non-deterministic. A single test tells you nothing. Run each case multiple times.
4. Set thresholds before you see the numbers.Decide what’s good enough in advance. Trust your criteria, or update them deliberately.