TokenPad
Evaluation

Prompt A/B Test Significance Calculator

Is your prompt actually better, or did you run 40 cases and get lucky?

Two variants with successes and trials

47 characters2 lines0 tokensor drop a file

A/B SignificanceExact
0Token change
Token change0no change
Input tokens0what you pasted
Output tokens0what you would send
Token cost of this result
Output tokens0
As input$0.00
× 100K requests$0.00

Everything on this page runs in your browser. Nothing you paste is transmitted, because there is no server here to transmit it to.

Result

Small evaluation sets lie constantly

On forty cases, a jump from 70% to 80% is well within what chance produces. Teams ship prompt changes on differences like that regularly, and half of them are shipping noise.

The confidence interval is the number to read. If it spans zero, you do not have evidence of a difference, however encouraging the point estimate looks.

Significant is not the same as worth shipping

A one-point improvement can be real and still not worth a prompt that costs thirty percent more to run. Statistical significance answers "is it real"; it says nothing about "is it worth it".

Put both variants through the prompt diff to see the cost side before deciding.

Frequently asked questions

How many cases do I need?
It depends on the effect size you care about. The calculator reports the requirement for the difference you observed — small differences need surprisingly large samples, which is itself useful to know before you start.
Can I stop early once it turns significant?
No. Checking repeatedly and stopping at the first significant result inflates the false positive rate substantially. Decide the sample size in advance and evaluate once.

More evaluation tools