Evaluation
Evaluation set
An evaluation set is a fixed collection of inputs with expected outputs, used to measure whether a change improved anything.
Also written: eval set, test set
Fifty cases covers the common paths for a narrow task; two hundred starts to catch the edges. Tag each case with a difficulty, because an aggregate score hides a change that improves easy cases and breaks hard ones.
Source cases from production failures first. Invented cases cluster around what you imagined, which is precisely the region the system already handles.
In practice
Below about fifty cases a single case flipping moves the score more than most real improvements do, so every result is noise. The highest-value cases are free: every bug that ever reached production. A regression suite that does not contain past regressions is not one, and a set that always passes is the same as having none.
Common questions
How many cases does an evaluation set need?
Fifty to detect whether something is badly broken, two hundred to detect the size of change a prompt edit usually produces. Below fifty, one case flipping moves the score more than most real improvements do.
Where do I get evaluation cases?
Production, and specifically production failures. Every bug that ever reached users is a free case, and a regression suite that does not contain past regressions is not one.
Measure it
- Evaluation Dataset BuilderCases with expected outputs and difficulty, exported three ways.
- Prompt A/B Test Significance CalculatorTwo-proportion z-test with confidence interval and required sample size.
- Confusion Matrix CalculatorFull metrics from four counts, plus the base rate warning accuracy needs.