9 tools
LLM Evaluation Tools
Find out whether a change actually improved anything.
Small evaluation sets lie constantly. On forty cases a jump from 70% to 80% is well within what chance produces, and teams ship prompt changes on differences like that every week.
These tools cover the whole loop: building a set, defining what good looks like, judging consistently, and checking that the difference you measured is real before you act on it.
- LLM Evaluation Rubric BuilderTurn criteria into a weighted rubric a judge can apply consistently.
- LLM-as-Judge Prompt BuilderBuild a judge prompt that does not just reward the longer answer.
- LLM Response ComparatorSee what one answer says that the other does not — numbers first.
- RAG Grounding CheckerFind the sentence the model added that the context never contained.
- Citation Extractor and ValidatorFind uncited claims, invalid indices, and sources you paid for and never used.
- Evaluation Dataset BuilderTurn a list of cases into a structured evaluation set with a difficulty mix.
- Confusion Matrix CalculatorPrecision, recall and F1 — with a warning when accuracy is lying to you.
- Inter-Rater Agreement Calculator (Cohen's Kappa)Find out whether your rubric is applicable before you automate it.
- Prompt A/B Test Significance CalculatorIs your prompt actually better, or did you run 40 cases and get lucky?
Which one you need
Build the dataset first. Everything else here is worthless without one, and the cases that matter are not invented — they come from real usage, and above all from every bug that ever reached production.
The splitter matters more than it looks: a holdout set you do not consult while iterating is the only defence against tuning to the test. Prompts get optimised toward whatever you look at, silently.
The rubric tools are for the stage where a person can no longer read every output. A judge scoring against named criteria produces a number with stable meaning between runs; "rate this one to five" does not.
The significance calculator is the one people skip and should not. On a hundred cases a three-point difference is usually noise, and a quarter of acting on noise is how a prompt ends up worse than where it started.
Questions
How many test cases do I need?
Fifty to tell whether something is badly broken, two hundred to detect the size of change a typical prompt edit produces. Below fifty a single case flipping moves the score more than most real improvements do, so every result is noise.
Can a model grade its own output?
It can grade output against a written rubric reasonably well, and it cannot be trusted before you measure its agreement with human judgement on about twenty cases. Below roughly 80% agreement you are optimising against the judge rather than the task.