Evaluation
LLM as judge
LLM-as-judge uses a language model to score or compare outputs against explicit criteria, in place of a human reviewer.
Pairwise comparison is more reliable than absolute scoring, because models compare better than they score in the abstract. It does not give you a quality level, and it scales quadratically with the number of candidates.
Validate a judge the way you would validate a second human reviewer: compute agreement against a human on a sample, and if it matches human-to-human agreement, the judge can replace the human on that task.
In practice
An uncalibrated judge measures its own preferences. Before trusting one, score twenty cases by hand and measure agreement; below roughly 80% you are optimising against the judge rather than the task. Run it at temperature zero, and give it a written rubric with named criteria — "rate one to five" produces a number with no stable meaning between runs.
Common questions
Is model-as-judge reliable?
For comparative judgements against a written rubric, reasonably. For absolute quality scores, less so. Measure agreement against human labels on twenty cases first — below roughly 80% you are optimising against the judge.
What temperature should a judge run at?
Zero. A grader that disagrees with itself between runs turns every comparison into a coin flip, and you will spend weeks chasing improvements that were sampling noise.
Measure it
- LLM-as-Judge Prompt BuilderJudge prompts with the verbosity and position bias countermeasures built in.
- LLM Evaluation Rubric BuilderWeighted criteria with score anchors and mandatory evidence quotes.
- Inter-Rater Agreement Calculator (Cohen's Kappa)Cohen's kappa with disagreement list. Below 0.6, fix the rubric first.