TokenPad

Models

Transformer

The transformer is the neural network architecture behind essentially every current language model, built around a mechanism called attention.

Its defining property is that it processes an entire input in parallel rather than one token at a time, which is what made training on internet-scale data practical.

That parallelism applies to reading, not writing. Generation is still sequential — each output token requires a full pass — which is precisely why output tokens cost four to six times more than input tokens.

In practice

The architectural fact with a billing consequence: attention cost grows with the square of sequence length. That is why a prompt twice as long can take more than twice as long to process, why time to first token climbs with input size, and why prompt caching — which skips recomputing a prefix — saves latency as well as money.

Common questions

Why does a longer prompt take disproportionately longer?

Attention cost grows with the square of sequence length. Doubling the input more than doubles the work, which is why time to first token climbs faster than input size and why trimming input helps latency as well as cost.

Do I need to understand transformers to build with LLMs?

Only three consequences of the architecture: attention is quadratic in length, position affects what gets used, and a fixed vocabulary makes token counts deterministic. The rest is not load-bearing for application work.

Measure it

  • AttentionAttention is the mechanism that lets a model weigh how much each token in the input matters when producing the next one.
  • Large language model (LLM)A large language model is a neural network trained to predict the next token in a sequence, which turns out to be enough to produce useful text, code and reasoning.
  • InferenceInference is running a trained model to produce output, as opposed to training, which is the process that produced the weights.

More in Models