TokenPad

11 tools

Data Format Tools for LLM Prompts

Convert payloads into the cheapest shape the model reads just as well.

The same records serialised six ways produce wildly different token counts. On flat tabular data the spread between cheapest and dearest is routinely three to one, and almost everyone sends the most expensive one because it is what their serialiser produces by default.

Every converter here shows the token change, because a format decision you cannot price is a format decision you cannot justify.

Which one you need

If something is failing to parse, start with the validator rather than the repairer. Knowing what is wrong is usually faster than automatically fixing something you have not diagnosed.

The conversion tools exist for the moment you change provider and discover that tool schemas, message shapes and result formats are all different. That conversion is where most migrations lose their time.

The minifiers matter more than they look on high-volume work. Pretty-printed JSON is roughly 2.2 characters per token against 4 for English prose, so the whitespace you cannot see is a real recurring line on the bill.

Format choice is itself a cost decision: the same data as CSV, JSON and YAML produces meaningfully different token counts, and at volume the difference is not a rounding error.

Questions

Which data format uses the fewest tokens?

For tabular data, CSV generally, because JSON repeats every key on every row. The gap widens with row count — at a thousand rows the repeated keys can be most of the payload. Measure with your own data rather than trusting the general rule.

Why does my JSON keep failing to parse?

Usually a markdown fence around it, a conversational preamble before it, or a trailing comma. Use the API’s schema enforcement rather than asking for JSON in the prompt, and keep a repair path anyway — enforcement is a strong constraint, not a guarantee.

Other categories