Harness Engineering

I The Gap

Parts 1 and 2 of this series gave you the vocabulary: failure modes, evaluation methods, the ship/hold decision, regression testing. You know what to measure and why it matters. But knowing that you should run evals on every change is very different from having the infrastructure that actually does it.

The gap between “I evaluate my AI feature” and “my AI feature is continuously evaluated” is an engineering problem. It’s the difference between a developer manually running 10 prompts in a playground and a pipeline that automatically scores 500 test cases across 5 runs each, compares the distribution against your last deploy, and blocks the merge if safety drops below threshold.

That infrastructure is the eval harness. And most teams don’t have one.

II Anatomy of an Eval Harness

An eval harness is a pipeline that runs automatically and answers one question: is this change safe to ship? It has five stages:

1. Test set. A curated collection of inputs with expected behaviors. Not just happy paths — edge cases, adversarial inputs, diversity cases (languages, tones, accessibility). Start with 30–50 cases. Grow it every time you find a bug in production. A good test set is a living document of everything your AI has ever gotten wrong.

2. Model run. Execute each test case against your current system. This isn’t just the model — it’s the full pipeline: retrieval, prompt assembly, model call, post-processing. Test the system as the user experiences it, not the model in isolation.

3. Scoring. Grade each output. Three options, roughly in order of cost and quality:

• Exact match / regex — fast, cheap, brittle. Good for structured outputs (JSON, classifications).
• LLM-as-judge — an LLM grades the output against a rubric. 80–90% agreement with humans at ~1/100th the cost. The fastest-growing approach, but you need to test the judge itself.
• Human review — gold standard for subjective quality. Reserve for a rotating sample, not every case.

4. Analysis. Compare scores against your baseline. Not individual outputs — distributions. Did accuracy shift? Did any failure mode spike? Did latency change? A single test run tells you almost nothing when the system is non-deterministic. You need to see the shape of the distribution.

5. Decision. Ship or hold, based on thresholds you set before seeing the results (Part 2 covered why). The harness surfaces the data; a human makes the call on ambiguous cases. Automate the clear passes and clear failures; escalate the gray zone.

III Testing Non-Deterministic Systems

Traditional software testing assumes stable behavior: same input, same output, every time. LLMs shatter this assumption. The same prompt can produce different outputs across runs, model updates, or even time of day. This isn’t a bug — it’s the architecture.

The practical consequence: a single test run tells you almost nothing. You need multiple runs per test case to see the distribution. If you run a case once and it passes, you know the model can produce a good answer. Run it 10 times and 3 fail? Now you know it produces a good answer 70% of the time. That’s a very different number to ship on.

How many runs? The standard practice is 5–10 runs per test case. Five gives you a rough signal. Ten gives you enough to calculate a meaningful confidence interval. More than that has diminishing returns unless you’re measuring a subtle regression.

Distinguishing variance from regression. If your accuracy is normally 92% ±2%, and today’s run shows 89%, is that a regression or a bad roll? Statistical significance matters here. Track your metrics over time to establish a baseline variance. A drop within the normal band is noise; a drop outside it is signal. The teams that skip this step either ship regressions they didn’t catch or block good changes because of random variance.

The cost model matters. 50 test cases × 10 runs = 500 model calls per eval. At $0.01 per call, that’s $5 per eval run. Run it on every PR and you’re spending maybe $50/week. Add LLM-as-judge scoring and double it. This is not expensive — it’s cheaper than one hour of a single engineer’s debugging time when a regression ships to production.

IV Observability

Eval harnesses catch problems before deploy. Observability catches them after. You need both, because no test set covers everything your users will do.

Log everything. Every production interaction should capture: the full prompt (system + user + retrieved context), the raw model response, latency, token counts, cost, and any tool calls. Not for every user forever — sample at a rate that gives you statistical coverage. 1–5% of production traffic is typical.

Score a sample. Run your LLM-as-judge or automated metrics on the sampled outputs. This gives you a continuous quality signal without reviewing every interaction. When the score drops, investigate. When it spikes, figure out what changed and protect it.

Alert on anomalies. Set thresholds for key metrics: if average safety score drops below 95%, if p99 latency exceeds 5 seconds, if cost-per-request doubles. These should page someone, not just log. The Replit incident — where an agent deleted a production database and generated fake records to cover it up — was only caught because someone happened to look. Don’t rely on luck.

Build the feedback loop. Every production failure that your observability catches should become a new test case in your eval harness. This is how your harness gets smarter over time. Garry Tan calls this the gap between “sophisticated agent evals” and “preventing agents from repeating failures” — the fix is explicit: every failure feeds back into the test set.

V The Complexity Ratchet

Here’s the counterintuitive truth about AI-assisted development: the faster you ship, the more rigor you need. Not less. More.

AI coding agents let you ship features at a pace that was impossible a year ago. But that velocity has a hidden cost. Every new feature, every agent-generated test, every deployed system exponentially increases your system’s fragility if not properly managed. Garry Tan calls this the complexity ratchet: AI agents have shattered the old speed-vs-quality tradeoff. Shipping fast with agents now requires 90%+ test coverage and rigorous quality gates — the opposite of “move fast and break things.”

Mario Zechner, after a year of shipping with coding agents, reflects on the same pattern: the temptation to “move fast” and the necessity of slowing down — because agent-generated code can compound mistakes across the codebase faster than humans can catch them.

And Aditya Agarwal pushes the point further: Silicon Valley conflates speed with ambition, when speed is often just impatience. The fast lane leads to mediocrity because velocity without direction is movement without progress.

The practical implication for your eval harness: as your deploy frequency increases, your eval coverage must increase proportionally. A team deploying once a week can get away with manual spot-checks. A team deploying five times a day with AI-generated code needs automated evals on every PR, continuous production monitoring, and a regression suite that grows with every failure. The ratchet only turns one direction.

VI Prompts as Infrastructure

Most teams treat prompts as strings in application code. This works until you need to answer: which version of the prompt is in production? What changed between v12 and v13? Can we roll back to yesterday’s prompt without redeploying the application?

Prompts are infrastructure. Treat them that way:

Version everything. System prompts, few-shot examples, tool descriptions, output format instructions — all versioned. A git diff should tell you exactly what changed in your AI behavior between any two deploys. This is the single most impactful practice for teams that skip it.

Separate prompt changes from code changes. A prompt tweak shouldn’t require a full application deploy. Store prompts in files, a config service, or a database — anywhere that can be updated independently. This lets you iterate on AI behavior without touching the codebase.

A/B test systematically. When you’re choosing between two prompts, don’t guess. Run both through your eval harness, compare the distributions, and pick the one that wins on the metrics you care about. In production, split traffic and measure. The eval harness you already built is the infrastructure for this.

Keep rollback instant. If a prompt change degrades quality, you need to revert in seconds, not minutes. This means the previous version is always one click away. Prompt rollback is your circuit breaker for AI behavior.