What Makes AI “Good”?

I The Evaluation Problem

Traditional software has clear pass/fail tests. 2+2 returns 4. Valid credentials produce a session token. The expected output is known, the actual output is deterministic, the comparison is binary.

AI is different. Ask an LLM to “write a friendly customer support email about a delayed shipment” and you get something that's… probably fine? Maybe great? Maybe subtly wrong in ways nobody notices until a customer screenshots it for Twitter. There's no single correct answer — just thousands of acceptable ones and a minefield of bad ones that look perfectly good at first glance.

Three properties make AI outputs uniquely hard to evaluate:

Non-determinism.Same prompt, different output every time — sometimes structurally, tonally, and factually different. You can't test once and move on. You need to reason about distributions, not individual results.

Subjectivity.“Good” depends on who's judging. A “friendly” email reads “too casual” to one reviewer. A “concise” summary feels “incomplete” to another. Inter-rater agreement on open-ended quality hovers around 60–75% — a number that should give any builder pause.

Multi-dimensionality.AI outputs can be accurate but unhelpful, helpful but unsafe, safe but boring, creative but wrong. Quality isn't a number — it's a vector. A medical chatbot and a creative writing tool have almost opposite quality criteria. Stanford's 2026 AI Index calls this the “jagged frontier”: Google's Gemini earned an International Math Olympiad gold medal but reads analog clocks correctly just 50.1% of the time. Gold-medal peaks and coin-flip valleys in the same model.

II Types of “Wrong”

Not all AI failures are created equal. Before you evaluate anything, you need a taxonomy of how outputs go wrong. Six failure modes every builder should know:

1. Hallucination— the model states fabricated information with total confidence. It invents statistics, cites papers that don't exist, describes features your company has never built. The danger: hallucinated content readsexactly like factual content. No hedging, no uncertainty flags — just fiction wearing a lab coat. The Penrose accounting benchmark showed why this matters for multi-step workflows: small per-step errors compound silently, so a model that's 95% accurate on each step can be wildly wrong by the end of a ten-step task.

2. Tone mismatch — correct information, wrong register. A legal compliance response written like a blog post. A customer apology that sounds like it was drafted by a committee of lawyers. The facts are right; the packaging undermines them.

3. Format errors — you asked for a bulleted list and got prose. Asked for JSON, got markdown. Asked for three options, got seven. Seems minor until downstream code tries to parse the output and crashes.

4. Unsafe outputs — harmful, biased, or inappropriate content. Ranges from subtle bias in hiring recommendations to overtly harmful instructions. The highest-stakes failure mode by reputational and legal risk. And the trend is going the wrong way: Stanford HAI documented 362 AI-related incidents in 2025, up 55% year-over-year, while responsible AI benchmark reporting among frontier developers remains spotty at best.

5. Refusals— the model declines a perfectly valid request out of excessive caution. A user asks about medication interactions and gets “I can't provide medical advice.” The mirror image of unsafe outputs — and equally damaging to user experience.

6. Partial answers — correct but incomplete. Answers two of three questions. Summarizes the first half and trails off. Gives the diagnosis but not the treatment. Technically right in the way that showing up to half your shift is technically showing up.

III Evaluation Methods

Four approaches to evaluating AI outputs, each with distinct tradeoffs. Knowing when to use which is one of the most valuable skills a builder can develop.

1. Human evaluation— the gold standard. Raters read outputs and score them against criteria. Humans catch what machines can't: a joke that falls flat, a recommendation that's culturally tone-deaf, a summary that misses the emotional core. The problem: $5–50 per judgment, inter-rater agreement around 60–75%, and a pace that makes your release cycle weep.

2. Automated metrics— fast, cheap, and shallow. BLEU and ROUGE measure text overlap against a reference answer. Exact match, regex patterns, and JSON validation handle structural checks. Excellent for “did it extract the right date?” Nearly useless for “is this a good summary?” A beautifully written paragraph can score poorly on ROUGE simply for choosing different words. The eval equivalent of judging a novel by its word count.

3. LLM-as-judge— the most exciting recent development. A stronger model evaluates a weaker model's output against a rubric you provide. Sounds circular, works surprisingly well — 80–90% agreement with human raters at ~1/100th the cost. The key is rubric precision. “Is this good?” produces unreliable scores. “Does the response contain only verifiable facts? Rate 1–5” produces consistent, actionable ones.

4. Rubric-based scoring — the glue that makes everything else work. Break quality into independently scorable dimensions: accuracy, helpfulness, tone, completeness. Each gets a clear definition and rating scale. Works with human raters, LLM judges, or both. When a score drops, you know which dimensiondegraded — not just that “quality went down somehow.”

IV The Eval Spectrum

Most teams start in the same place: vibes. Someone runs a few prompts, reads the outputs, and says “looks good to me.” Cherry-picked examples get shared in Slack. Everyone nods. The feature ships. This is the evaluation equivalent of checking your parachute by looking at it and thinking “yeah, that's probably fine.”

Vibes work for prototypes. They fail at everything else. The problem isn't human judgment — it's that unstructured, ad-hoc judgment is inconsistent, unscalable, and impossible to reproduce. You can't tell if this week's model is better than last week's when your evaluation was “Sarah looked at 10 outputs on Tuesday and felt pretty good about them.”

V What's Next

You now have the evaluation mindset: why AI quality is hard to measure, the failure modes to watch for, the methods available, and the maturity spectrum from vibes to production monitoring.

In Part 2: Evals in Practice, we get hands-on — building test sets that catch regressions, strategies for handling non-determinism, and the framework for making ship-or-hold decisions when your eval results are ambiguous. Because in the real world, they almost always are.