Evaluation
Benchmark
A benchmark is a standard test set used to compare models — useful for a rough shortlist and close to useless for predicting performance on your task.
Public benchmarks leak into training data over time, which inflates scores without improving capability. A model topping a leaderboard may simply have seen the answers.
They also measure the average of many tasks, and you have one task. A model two points lower overall can be substantially better at yours.
In practice
Published scores are run on public sets the model may have trained on, and reported by parties with an interest in the result. Neither makes them worthless, but neither makes them a prediction about your data. Thirty real cases from your own traffic tell you more about your task than any leaderboard position.
Common questions
Should I choose a model by benchmark scores?
Not on their own. They are run on public sets the model may have trained on and reported by parties with an interest in the result. Thirty cases from your own traffic predict your outcome better.
What is contamination?
Benchmark questions appearing in training data, which turns a reasoning test into a recall test. It is difficult to rule out for any public set, and it is the main reason scores drift upward faster than capability.
Measure it
- Evaluation Dataset BuilderCases with expected outputs and difficulty, exported three ways.
- LLM Model FinderSet a budget and a context requirement, get the shortlist. Sorted by your own input-to-output ratio.
- Prompt A/B Test Significance CalculatorTwo-proportion z-test with confidence interval and required sample size.