Models
Quantization
Quantization stores model weights at lower numerical precision — 8-bit or 4-bit instead of 16-bit — to reduce memory and increase speed.
It is what makes running a capable model on consumer hardware possible. An 8-bit quantization roughly halves memory against 16-bit, and 4-bit roughly quarters it.
Quality loss is usually small and task-dependent. It tends to show up first on reasoning and long-context tasks rather than on simple classification.
In practice
Quantization is what makes a model runnable on hardware you own — 8-bit roughly halves memory against 16-bit, 4-bit halves it again, with quality loss that is small on most tasks and not small on precise ones like arithmetic or code. It is the main lever when the constraint is a GPU you already have rather than a per-token price.
Common questions
How much quality does quantization cost?
Small on most tasks at 8-bit, noticeable at 4-bit, and worst on precise work — arithmetic, code, exact recall. The right test is your own evaluation set at each level, because the loss is task-dependent.
When does quantization matter?
When you run models yourself and the constraint is GPU memory you already own. It has no bearing on API pricing, where the provider has already made that choice for you.