Evaluation
Regression testing
Regression testing re-runs a fixed evaluation set after every change, so you find out what a prompt edit broke as well as what it fixed.
Prompts are code with no type system and no compiler. A change that fixes one case routinely breaks two others, and without a fixed set nobody notices until a user does.
Tag cases by difficulty. An aggregate score hides a change that improves easy cases while breaking hard ones, which is the most common shape of a bad prompt edit.
In practice
Prompt edits routinely fix the case in front of you and break two you were not looking at. A fixed set run before and after is the only thing that catches it, and score differences need a significance check before being believed — on a hundred cases a three-point move is usually noise, and acting on noise is how prompts drift worse over a quarter.
Common questions
How do I know a prompt change actually helped?
Run a fixed set before and after and test the difference for significance. On a hundred cases a three-point move is usually noise, and acting on noise is how prompts drift worse over a quarter.
When should the evaluation set run?
On every prompt change, which means it has to be one command. Anything harder gets run before launches and not before edits, which is exactly backwards.
Measure it
- Evaluation Dataset BuilderCases with expected outputs and difficulty, exported three ways.
- Prompt Diff and Token ComparisonBefore and after, with the token delta priced. The missing half of every prompt rewrite.
- Prompt A/B Test Significance CalculatorTwo-proportion z-test with confidence interval and required sample size.