LLM-as-Judge Done Right | Airtribe Resources

I Why Judge?

An answer sheet flows in from the left; a samosa inspects it through a magnifying glass and assigns a verdict shown on the right as a bold orange score, 7 out of 10. — LLM-as-judge: one model marks another's answer against a standard — grading at machine speed instead of waiting on humans.

Part 1 introduced four evaluation methods: human eval, automated metrics, LLM-as-judge, and rubric-based scoring. In practice, most teams land on LLM-as-judge as their workhorse. Here's why.

Human evaluation doesn't scale.At $5–50 per judgment, evaluating 500 test cases costs $2,500–25,000 per eval run. Run that on every PR and you're bankrupt by Tuesday. Human eval is the gold standard for calibrating your system, not for running it continuously.

Automated metrics are too shallow. BLEU, ROUGE, exact match — these work for structured outputs but fall apart for open-ended generation. A perfectly good email summary can score 0.2 on ROUGE because it used different words than the reference. Metrics that compare strings miss everything about meaning, tone, and completeness.

LLM-as-judge fills the gap. 80–90% agreement with human raters at roughly 1/100th the cost. Fast enough to run on every PR. Nuanced enough to evaluate open-ended outputs. The catch: that 80–90% agreement only holds when you do it right. A lazy judge prompt produces expensive random numbers.

Key Insight

LLM-as-judge isn't a replacement for human evaluation. It's a proxythat you calibrate against human judgment. Humans set the standard (what does a 5/5 look like?). The judge scales it (score 10,000 outputs against that standard). If you skip the human calibration step, you're just measuring consistency, not quality.

II Anatomy of a Judge

A well-built LLM judge has four components. Miss any one and the scores become unreliable.

1. The rubric.This is the entire game. A rubric defines what you're measuring, how to score it, and what each score level means. "Is this response good?" is not a rubric. "Rate factual accuracy 1–5, where 1 = contains fabricated claims, 3 = mostly accurate with minor errors, 5 = every claim is verifiable" is a rubric. The more specific your rubric, the more consistent your scores.

2. The grading prompt. The system prompt that turns the LLM into a judge. It includes: the rubric, the scoring scale, instructions to evaluate onlywhat the rubric asks (not general quality), and a requirement to provide evidence for the score. A judge that says "4/5" without explaining why is useless for debugging.

3. The evidence requirement.Always require the judge to cite specific parts of the output that justify its score. "I gave this a 2 because the response claims the product launched in 2023, but the source document says 2024" is actionable. "I gave this a 2 because it didn't seem accurate" is not.

4. Structured output. Have the judge return JSON, not prose. {"accuracy": 4, "completeness": 3, "tone": 5, "evidence": "..."} is parseable and aggregatable. A paragraph of feedback requires another LLM call to extract the scores.

Builder Tip

One dimension per judge call.Don't ask a judge to score accuracy, completeness, tone, and safety in a single prompt. Scores bleed into each other — a factually wrong but well-written response gets inflated accuracy scores because the judge is influenced by the good writing. Score each dimension in a separate call. It costs more but produces dramatically better data.

III Strengths & Blindspots

LLM judges are not universally reliable. They have specific strengths and specific blindspots, and knowing which is which determines whether your eval system works.

Strong at:

Factual accuracy — verifying claims against provided source material. The judge can cross-reference the output with the retrieval results and flag discrepancies.
Format compliance — checking whether the output follows structural requirements (JSON schema, word count, required sections).
Consistency — given the same rubric, an LLM judge produces more consistent scores than a rotating pool of human raters. No bad days, no lunch fatigue.
Relative comparison— "which of these two responses is better?" produces more reliable results than "rate this response 1–5." Pairwise comparison reduces calibration drift.

Weak at:

Novelty and creativity — LLM judges reward fluent, conventional responses and penalize surprising but effective ones. If your product values originality, the judge will systematically underrate your best outputs.
Cultural nuance — humor, formality norms, and communication styles vary across cultures. A judge trained on English-dominant data will misjudge outputs intended for other cultural contexts.
Catching its own failure modes— if the judge uses the same model family as the generator, they share the same blindspots. Both might consider a confident hallucination "well-written and plausible."
Verbosity bias — LLM judges consistently rate longer, more detailed responses higher than concise ones, even when brevity is better. A judge evaluating customer support replies will prefer the 200-word essay over the 30-word answer that actually solves the problem.

Analogy

An LLM judge is like a teaching assistant grading papers with a rubric. Given clear criteria, they're remarkably consistent and fast. But they grade to the rubric, not beyond it. If the rubric doesn't capture what matters, the TA will confidently assign scores that miss the point entirely. The rubric is the ceiling of the judge's intelligence.

IV The Circular Problem

The obvious objection: using an LLM to judge an LLM is circular. If the model can't tell its own output is wrong, how can it tell another model's output is wrong?

The objection is half right. Here's when it's valid and when it's not:

Circular (avoid):

Same model judges its own output. Claude evaluating Claude with no rubric is just the model saying "yeah, that sounds like something I'd say."
Judging on dimensions the model is known to be bad at. If your model hallucinates, using it to detect hallucinations in another model's output is asking the blind to lead the blind.
No ground truth for calibration. If you've never had humans score a subset, you have no way to know if the judge's scores mean anything.

Not circular (use confidently):

Different model judges a different model. A GPT-class model judging Claude, or Opus judging Haiku. Different training data, different failure modes, different blindspots. Cross-model judging catches failures that same-model judging misses.
Rubric-constrained evaluation. A well-crafted rubric with specific criteria and evidence requirements forces the judge to evaluate specific claims, not vibes. The rubric turns an open-ended question into a verification task, which LLMs are much better at.
Verification is easier than generation.This is the fundamental reason LLM-as-judge works. Checking whether a response correctly cites a source document is much simpler than generating the response in the first place. The judge doesn't need to be smarter than the generator — it needs to be good at checking specific properties.

Key Insight

Verification is easier than generation.You don't need a PhD to grade a PhD thesis. You need clear criteria, access to references, and enough expertise to check claims. LLM judges work for the same reason — checking whether "the response only contains claims supported by the retrieved documents" is a simpler task than generating the response.

V Calibrating Your Judge

A judge is only as good as its calibration. Here's how to know if yours is reliable.

Step 1: Human baseline.Have 2–3 human raters score 50–100 outputs using the same rubric the judge will use. Measure inter-rater agreement (humans disagreeing with each other). If your human raters only agree 70% of the time, your judge can't be expected to do better — that's the ceiling.

Step 2: Judge-vs-human agreement.Run the same 50–100 outputs through your LLM judge. Compare judge scores to human scores. You want agreement at or above the human-vs-human baseline. If humans agree 75% of the time and the judge agrees with humans 73% of the time, that's a well-calibrated judge.

Step 3: Failure analysis. Look at the cases where the judge and humans disagree. Are there patterns? Maybe the judge consistently overscores long responses (verbosity bias). Maybe it underscores creative writing. These patterns tell you where to add rubric specificity or where to fall back to human review.

Step 4: Ongoing spot-checks. Calibration drifts. Model updates change behavior. Every week, pull 20–30 judge-scored outputs and have a human verify the scores. If agreement drops below your threshold, recalibrate the rubric. This is cheap insurance against silent degradation.

Builder Tip

Pairwise comparison beats absolute scoring.Instead of "rate this response 1–5," try "which of these two responses is better on [dimension]?" Pairwise judgments have higher inter-judge agreement, sidestep calibration drift (no need to agree on what "3" means), and map naturally to A/B testing. The downside: you need O(n²) comparisons for n outputs, so it's best for small sets or tournament-style brackets.

VI Judge at Scale

Once your judge is calibrated, plug it into the eval harness from Part 3. Here's what changes at scale.

Cost model.A typical judge call costs $0.005–0.02 per output (depending on model and output length). At 500 test cases × 5 runs × 3 dimensions scored separately = 7,500 judge calls per eval. At $0.01 each, that's $75 per eval run. Run it on every PR across a 10-person team and you're at maybe $500–1,000/month. Compare that to one production incident caused by a regression you didn't catch.

Async scoring.Judge calls don't need to block the pipeline. Run your model outputs in parallel, queue the judge calls, and aggregate scores asynchronously. The developer sees "eval running" on the PR and gets results in minutes, not blocking their workflow.

Escalation to human review.Build an escalation path for outputs the judge is uncertain about. If the judge scores accuracy at 2/5 but completeness at 5/5, that's a weird combination worth a human look. If the judge's confidence is low (scores cluster around the middle of the scale), escalate. Reserve your human review budget for the cases the judge flags, not random sampling.

Version your rubrics.Rubrics evolve as your product evolves. When you change a rubric, scores before and after aren't comparable — treat it like a new metric. Version rubrics alongside prompts (Part 3), and re-run the calibration step when rubrics change.

Takeaway

LLM-as-judge is the eval method that scales. The four things that make it work: a specific rubric (vague rubrics produce random numbers), one dimension per call (dimensions bleed into each other), evidence requirements (a score without reasoning is noise), and human calibration(you need a ground truth to validate against). Skip any of these and you're paying for confident nonsense.