84 terms
AI glossary
Definitions that say something a dictionary would not: what the thing costs, what it trades off, and the mistake almost everyone makes with it. Every term that can be measured links to the tool that measures it.
Tokens
- Byte-pair encoding (BPE)Byte-pair encoding builds a tokenizer vocabulary by repeatedly merging the most frequent adjacent pair of symbols in a corpus until it reaches a target size.
- Context rotContext rot is the degradation in recall and instruction-following that occurs as a model’s context window fills up.
- Context windowThe context window is the total token budget for a single request, covering the system prompt, conversation history, tool definitions, documents and the response the model generates.
- o200k_baseo200k_base is the byte-pair encoding used by GPT-4o and later OpenAI models, with roughly twice the vocabulary of the cl100k_base it replaced.
- TokenA token is an entry in the fixed vocabulary a language model was trained with — the unit it reads, and the unit you are billed in.
- TokenizerA tokenizer is the frozen vocabulary and merge rules that convert text into tokens. Each provider trains its own, so the same text costs different amounts on different models.
Cost
- Batch APIA batch API processes requests asynchronously within a latency window of hours, in exchange for a discount that is commonly around 50%.
- Cached inputCached input is prompt content the provider has already processed and retained, billed at roughly a tenth of the base input rate.
- Input tokensInput tokens are everything you send to the model: system prompt, conversation history, tool definitions, documents and the user message.
- Model routingModel routing sends each request to the cheapest model that can handle it, escalating to a more capable one only when needed.
- Output tokensOutput tokens are what the model generates. They typically cost four to six times more than input tokens.
- Rate limitA rate limit caps how much you may send across all requests in a period, usually expressed as tokens per minute and requests per minute.
- Reasoning tokensReasoning tokens are the internal thinking a reasoning model generates before its answer. They are billed as output and never returned to you.
- ThroughputThroughput is how much work you can push through per unit of time, bounded by your rate limits and your concurrency.
- Time to first token (TTFT)Time to first token is the delay between sending a request and the first token arriving — dominated by the time spent processing your input.
Prompting
- Chain of thoughtChain-of-thought prompting asks the model to work through intermediate reasoning before giving an answer, which improves accuracy on multi-step problems.
- DelimiterA delimiter marks where one section of a prompt ends and another begins, usually with XML-style tags.
- Few-shot promptingFew-shot prompting supplies a handful of worked examples in the prompt so the model infers the pattern rather than being told it.
- Prompt engineeringPrompt engineering is the practice of structuring the text sent to a language model so that its output is reliable enough to build on.
- Prompt templateA prompt template is a prompt with placeholders, filled with real values at request time.
- System promptA system prompt is the standing instruction sent with every request, defining the model’s role, task, constraints and output format.
- Zero-shot promptingZero-shot prompting asks the model to perform a task with instructions only, supplying no worked examples.
Agents
- AI agentAn AI agent is a loop in which a model receives a request, decides whether to call a tool, receives the result, and repeats until it can answer.
- AI skillA skill is a file packaging a reusable capability: a name, a description that decides when it loads, and a body of instructions loaded only once it does.
- CompactionCompaction rewrites accumulated conversation history into a compact state summary so a long session can continue without exceeding its budget or window.
- Model Context Protocol (MCP)MCP is an open protocol that lets an AI client connect to servers exposing tools, resources and prompts through a standard interface.
- Multi-agent systemA multi-agent system splits work across several agents with different roles, tools or models, passing control between them.
- ReActReAct is an agent pattern where the model alternates between reasoning about what to do next and acting by calling a tool.
- Tool callingTool calling lets a model request that your code run a named function with structured arguments, then continue with the result.
RAG
- Chunk overlapChunk overlap repeats a portion of each chunk in the next one, so a sentence landing on a boundary appears whole in at least one of them.
- ChunkingChunking splits documents into passages small enough to embed usefully and large enough to be interpretable on their own.
- Context stuffingContext stuffing is filling a large context window with everything that might be relevant, instead of retrieving what is.
- Cosine similarityCosine similarity measures the angle between two embedding vectors, giving a score from -1 to 1 for how similar their meanings are.
- EmbeddingAn embedding is a vector of numbers representing the meaning of a piece of text, so that similar meanings sit near each other in that space.
- Embedding dimensionsThe dimension count is the length of an embedding vector — commonly 384, 768, 1536 or 3072 — and it drives storage and search cost directly.
- Hybrid searchHybrid search combines keyword matching with semantic search, then merges the two result sets into one ranking.
- Knowledge baseA knowledge base is the corpus a retrieval system searches — the documents, chunks and embeddings that ground its answers.
- RerankingReranking scores a wide set of retrieved candidates with a more precise model and keeps only the best few for the prompt.
- Retrieval-augmented generation (RAG)RAG finds passages relevant to a question and places them in the prompt, so the model answers from supplied material rather than from training data.
- Semantic searchSemantic search finds text by meaning rather than by keyword, by comparing embeddings instead of matching words.
- Top-k retrievalTop-k is how many chunks you retrieve and place in the prompt. It is the single largest cost lever in a retrieval system.
- Vector databaseA vector database stores embeddings and finds the nearest ones to a query vector, usually with an approximate nearest-neighbour index.
Models
- AttentionAttention is the mechanism that lets a model weigh how much each token in the input matters when producing the next one.
- DistillationDistillation trains a smaller model to imitate a larger one, producing something much cheaper to run that retains most of the capability on a narrower range of tasks.
- Fine-tuningFine-tuning continues training a base model on your own examples so it adopts a format, tone or behaviour without being told each time.
- InferenceInference is running a trained model to produce output, as opposed to training, which is the process that produced the weights.
- Instruction tuningInstruction tuning is additional training that teaches a base model to follow instructions rather than merely continue text.
- JSONLJSONL is a file format with one JSON object per line, used for fine-tuning datasets and batch requests.
- Knowledge cutoffThe knowledge cutoff is the date after which a model has no training data, and therefore no reliable knowledge of events.
- Large language model (LLM)A large language model is a neural network trained to predict the next token in a sequence, which turns out to be enough to produce useful text, code and reasoning.
- Mixture of experts (MoE)A mixture-of-experts model contains many specialised sub-networks and activates only a few per token, so it has the capability of a large model at the compute cost of a smaller one.
- Model deprecationModel deprecation is a provider retiring a model version, after which requests to it fail or are silently redirected.
- MultimodalA multimodal model accepts more than one kind of input — commonly text plus images, sometimes audio or video.
- Open weightsAn open-weights model has downloadable parameters, so you can run it on your own hardware instead of calling somebody’s API.
- ParametersParameters are the learned weights inside a model. The count — 7B, 70B, 405B — is a rough proxy for capability and a direct driver of cost and latency.
- QuantizationQuantization stores model weights at lower numerical precision — 8-bit or 4-bit instead of 16-bit — to reduce memory and increase speed.
- RLHFRLHF trains a model against human preferences between candidate outputs, rather than against a fixed correct answer.
- TransformerThe transformer is the neural network architecture behind essentially every current language model, built around a mechanism called attention.
Evaluation
- BenchmarkA benchmark is a standard test set used to compare models — useful for a rough shortlist and close to useless for predicting performance on your task.
- Evaluation setAn evaluation set is a fixed collection of inputs with expected outputs, used to measure whether a change improved anything.
- GroundingGrounding is the property of an answer being supported by supplied source material rather than by the model’s training data.
- HallucinationA hallucination is model output that is fluent, confident and not supported by the source material or by fact.
- LLM as judgeLLM-as-judge uses a language model to score or compare outputs against explicit criteria, in place of a human reviewer.
- Precision and recallPrecision is what share of the things you flagged were right; recall is what share of the things that existed you caught.
- Regression testingRegression testing re-runs a fixed evaluation set after every change, so you find out what a prompt edit broke as well as what it fixed.
- Self-consistencySelf-consistency runs the same prompt several times at non-zero temperature and takes the most common answer.
Safety
- AlignmentAlignment is the work of making a model behave in accordance with human intent — helpful, honest, and declining what it should decline.
- API keyAn API key authenticates your requests to a provider and, in most cases, is the only thing standing between an attacker and your billing account.
- Data residencyData residency is the guarantee that your requests are processed within a specific geography, usually required by contract or regulation.
- GuardrailsGuardrails are the rules constraining what a model or agent may do, stated in the prompt and — where it matters — enforced outside the model.
- Human in the loopHuman in the loop means a person approves or corrects an agent’s action before it takes effect.
- JailbreakA jailbreak is an attempt to make a model bypass its own safety training, usually through role-play framing, mode-switch phrasing or hypothetical scenarios.
- PIIPII is data that identifies a person — names, emails, phone numbers, addresses, identifiers — and it should not reach an API you do not control without a decision behind it.
- Prompt injectionPrompt injection is an attack where text the application inserts into a prompt — a document, a user upload, a tool result — contains instructions the model then follows.
- Red teamingRed teaming is deliberately attacking your own system — with injections, jailbreaks and edge cases — to find failures before someone else does.
Parameters
- LogprobsLogprobs are the model’s reported confidence for each token it chose, and for the alternatives it considered.
- max_tokensmax_tokens caps how many tokens the model may generate in its reply. It is a ceiling on output only, subtracted from the same context window the input occupies.
- SeedA seed fixes the random starting point of sampling, so the same request produces the same output.
- Stop sequenceA stop sequence is a string that halts generation the moment the model produces it.
- StreamingStreaming sends tokens to the client as they are generated rather than waiting for the complete response.
- Structured outputStructured output constrains generation so the response conforms to a schema, rather than requesting a format and hoping.
- TemperatureTemperature scales the model’s output probabilities before sampling: below one sharpens the distribution towards the likeliest token, above one flattens it.
- Top-pTop-p keeps only the most probable tokens that together make up a given share of the probability mass, and samples from those.
- TruncationTruncation is generation stopping because a limit was reached rather than because the model finished.
How to use this
Every entry opens with a one-sentence answer, then two or three paragraphs of substance, then the thing that catches people out — because that last part is usually what you were actually looking for.
Terms that describe something measurable link straight to one of the 102 tools. Reading that a context window includes the model’s own output is useful; seeing how much of yours is left is more so.
If you are starting from zero, read token first — everything else on this site is denominated in them. Then context window and system prompt, which are where most production problems begin.