Glossary

Short, plain-language definitions of the terms used across the articles, each linked to the paper, standard or documentation that defines it. In an article, terms with a dotted underline show their definition when you hover over, tap or focus them.

3

3FS: DeepSeek's open-source distributed file system on SSDs and RDMA networks, used among other things to keep KV caches for reuse across requests.Source: 3FS (DeepSeek)Explained in Case Study: DeepSeek's Inference Stack and DeepInfer

8

8-bit optimizer: An optimizer that stores its running statistics, such as Adam's two moments, in 8 bits with one scale per small block of values, cutting their memory about fourfold with little loss in accuracy.Source: 8-bit Optimizers via Block-wise Quantization (Dettmers et al., 2022)Explained in Build an LLM From Scratch: Fitting Bigger Models in Memory

A

A2A: Agent2Agent Protocol: an open protocol for independent AI agents to discover each other and exchange tasks, messages and results over JSON-RPC, gRPC or HTTP, without sharing their internals.Source: A2A Protocol SpecificationExplained in Multi-Agent Systems: Orchestrators, Sub-agents and A2A, Measured
Acceptance rate: In speculative decoding, the fraction of drafted tokens the target model keeps. It sets how many tokens each verification pass produces on average.Source: Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Accessibility tree: The structured description of a page (roles, names, states) that browsers build for screen readers. Browser agents can read it as text instead of pixels.Source: Browser use tool (Claude API docs)Explained in Computer-Use and Browser Agents: Pixels Versus the Page
ACDCAutomatic Circuit DisCovery: An algorithm that finds a circuit by deleting connections between model components whose removal barely changes the output.Source: Towards Automated Circuit Discovery (Conmy et al., 2023)Explained in Circuits and Attribution Graphs, Traced on a Laptop GPU
ACLAccess control list: The record of which users or groups may see a document. In RAG it must be enforced inside retrieval, so nobody gets answers drawn from documents they cannot open.Source: Access control list (NIST glossary)Explained in Designing a RAG System for 10 Million Documents
Activation checkpointing: Keeping only some intermediate values (usually each layer's input) during the forward pass and recomputing the rest during the backward pass, trading extra compute for much less memory.Source: Reducing Activation Recomputation in Large Transformer Models (Korthikanti et al., 2022)Explained in Backpropagation by Hand: From the Chain Rule to Autograd
Activation function: A fixed nonlinear function applied to a neuron's weighted sum. Without one, stacked layers would collapse into a single linear layer.Source: Deep Learning, Chapter 6 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
Activation memory: Memory for the tensors a training forward pass must keep so the backward pass can compute gradients. It grows with batch size and sequence length.Source: Reducing Activation Recomputation in Large Transformer Models (Korthikanti et al., 2022)Explained in Backpropagation by Hand: From the Chain Rule to Autograd
Activation patching: Copying an internal activation from one run into another, such as from a clean prompt into a corrupted one, to measure how much that activation causes the output.Source: Locating and Editing Factual Associations in GPT (Meng et al., 2022)Explained in Activation Patching and Causal Tracing, Measured in GPT-2
Active parameters: The parameters a mixture-of-experts model actually uses for one token. They set compute per token, while the total parameter count sets memory.Source: Mixtral of Experts (Jiang et al., 2024)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
Adam: An optimizer that keeps running averages of each parameter's slope and squared slope, giving every parameter its own step size. AdamW is a common variant.Source: Adam: A Method for Stochastic Optimization (Kingma & Ba, 2014)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
Adaptive thinking: A Claude API mode in which the model decides whether and how much to reason before each reply, steered by an effort setting.Source: Effort (Claude API docs)Explained in What an AI Agent Is: The Loop, Measured
Affine map: A linear map followed by a shift, Wx + b. A neural network layer before its activation function is affine.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Matrices as Transformations, and Why Neural Networks Need a Bend
Agent Card: A JSON document an A2A agent publishes at /.well-known/agent-card.json describing its identity, skills, endpoints and authentication requirements.Source: A2A Protocol SpecificationExplained in Multi-Agent Systems: Orchestrators, Sub-agents and A2A, Measured
Agent harness: The code around a model that supplies its tools, runs the loop, manages its context and enforces limits such as step and cost caps.Source: How Claude Code works (Claude Code docs)Explained in What an AI Agent Is: The Loop, Measured
Agent-computer interface: The commands an agent uses to act on a computer and the format of the feedback it gets back, such as its search, file viewer and edit tools.Source: SWE-agent (Yang et al., 2024)Explained in Coding Agents: How They Fix Real Bugs, Measured
AgentDojo: A benchmark that measures how well tool-using agents complete tasks, and resist prompt injections, in simulated environments such as email, banking and travel.Source: AgentDojo (Debenedetti et al., 2024)Explained in Defending Tool-Using Agents Against Prompt Injection by Design
Agentic loop: The cycle an agent repeats: call the model, run any tools it requests, send back the results, and call it again until it stops asking for tools.Source: How tool use works (Claude API docs)Explained in What an AI Agent Is: The Loop, Measured
AI agent: A system in which a language model runs in a loop: it chooses tools to call, reads their results, and decides when the task is done.Source: Building effective agents (Anthropic, 2024)Explained in What an AI Agent Is: The Loop, Measured
ALiBiAttention with Linear Biases: Attention with Linear Biases: instead of position vectors, each attention head subtracts a fixed slope times the distance between query and key from the score, so farther tokens get less attention.Source: Train Short, Test Long (Press et al., 2021)Explained in Positional Encoding: How Transformers Know Word Order
Alignment tax: Performance lost on some tasks as a side effect of alignment training.Source: Training language models to follow instructions with human feedback (Ouyang et al., 2022)Explained in From Base Model to Assistant: SFT, RLHF and DPO
All-gather: A collective operation in which every GPU contributes a piece and ends up with all the pieces. FSDP uses it to rebuild sharded weights before each layer.Source: NCCL User Guide: Collective Operations (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
All-reduce: A collective operation that adds up a tensor across a group of GPUs and gives every GPU the total. Tensor parallelism needs two per Transformer block.Source: NCCL User Guide (NVIDIA)Explained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
All-to-all: A collective operation in which every GPU sends a different piece of data to every other GPU. MoE layers use it to dispatch tokens to experts and combine the results.Source: NCCL User Guide (NVIDIA)Explained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
amax: The largest absolute value in a tensor or block, used to set its quantization scale. For FP8 E4M3, scale = amax ÷ 448.Source: Transformer Engine: Using FP8 and FP4 (NVIDIA)Explained in Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
ANNApproximate nearest neighbor search: Finding the vectors closest to a query without comparing it against every stored vector. It gives up a little recall for search that is orders of magnitude faster.Source: The Faiss library (Douze et al., 2024)Explained in Designing a RAG System for 10 Million Documents
Annealing: The final phase of pretraining, where the learning rate drops to near zero while the highest-quality data is upsampled.Source: The Llama 3 Herd of Models (Llama Team, Meta, 2024)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Antipodal pair: Two features stored as exactly opposite directions so they share one dimension, each using half of it.Source: Toy Models of Superposition (Elhage et al., 2022)Explained in Superposition: How Models Store More Features Than Neurons
APGRAverage performance gap recovered: How much of the quality gap between a cheap and a strong model a router recovers, averaged across different shares of traffic sent to the strong model.Source: RouteLLM: Learning to Route LLMs with Preference Data (Ong et al., 2024)Explained in Routing Requests Between a Cheap and a Strong Model
Approval fatigue: When people approve permission prompts so often that they stop reading them, so the prompt no longer protects anything.Source: Making Claude Code more secure and autonomous with sandboxing (Anthropic)Explained in Sandboxing and Permissions for AI Agents, Measured
Argmax: The position or item where a list reaches its largest value. Softmax at a temperature near 0 approaches a one-hot argmax.Source: Deep Learning, chapter 6 (Goodfellow, Bengio & Courville)Explained in Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely
Arithmetic coding: Lossless compression that narrows an interval by each symbol's predicted probability, using about minus log2 q bits per symbol. With an LLM as the predictor, lower loss means smaller files.Source: Language Modeling Is Compression (Delétang et al., 2023)Explained in Entropy, Cross-Entropy and Perplexity
Arithmetic intensity: How many calculations a piece of work does per byte it reads from memory. Below the hardware’s balance point the work is memory-bound; above it, compute-bound.Source: Making Deep Learning Go Brrrr From First Principles (Horace He)Explained in How LLM Inference Works: Prefill, Decode and the Memory Wall
ASRAttack success rate: The share of attack attempts, such as prompt injections, that get the system to do what the attacker wanted.Source: AgentDojo (Debenedetti et al., 2024)Explained in Defending Tool-Using Agents Against Prompt Injection by Design
Atomic operation: A read-modify-write the hardware performs indivisibly, so threads updating one address don't lose updates. Float atomics add in a varying order, so results can differ from run to run.Source: CUDA Programming Guide: C++ language extensions (NVIDIA)Explained in Reductions and Normalization Kernels: Summing Across Threads
Attention head: One independent copy of the attention calculation, with its own query, key and value matrices. Models run many heads side by side in every layer.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Attention Explained: Queries, Keys and Values
Attention sink: Early tokens, usually the first, that receive a large share of attention whatever their meaning, because softmax forces each head's weights to add up to 1.Source: Efficient Streaming Language Models with Attention Sinks (Xiao et al., 2023)Explained in Attention Explained: Queries, Keys and Values
Attribution graph: A graph of which features influenced which, and the final output, on one prompt, computed through an interpretable replacement model.Source: Circuit Tracing (Ameisen et al., 2025)Explained in Circuits and Attribution Graphs, Traced on a Laptop GPU
Attribution patching: A gradient-based estimate of activation patching that scores every component from one forward and backward pass, as (clean minus corrupted activation) times the gradient.Source: Attribution Patching (Nanda, 2023)Explained in Circuits and Attribution Graphs, Traced on a Laptop GPU
AUROC: Area under the ROC curve: the chance that a randomly chosen positive example scores higher than a randomly chosen negative one. 0.5 is chance and 1 is perfect.Source: An introduction to ROC analysis (Fawcett, 2006)Explained in Linear Probes: Reading What a Model's Activations Encode
Autocast: A PyTorch context that runs selected operations, such as matrix multiplications, in a lower-precision format like BF16 while keeping precision-sensitive ones in FP32.Source: Automatic Mixed Precision package (PyTorch)Explained in Build an LLM From Scratch: Writing the Model
Autograd: Software that records the operations a program performs and applies the chain rule to compute exact gradients automatically. PyTorch's is called autograd.Source: Automatic differentiation in machine learning: a survey (Baydin et al., 2015)Explained in Derivatives, Gradients and the Chain Rule
Autotuning: Compiling several variants of a kernel (tile sizes, warps, pipeline stages) and timing each on the real input shape to pick the fastest.Source: triton.autotune (Triton docs)Explained in Triton: Writing GPU Kernels in Python, Measured
Auxiliary-loss-free balancing: DeepSeek's MoE load-balancing method: a per-expert bias, nudged after each step, is added to router scores only when choosing experts, instead of adding a balancing loss.Source: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts (Wang et al., 2024)Explained in Counting: Parameters, FLOPs, Memory and Scaling Laws
AWQActivation-aware weight quantization: A 4-bit weight quantization method that protects the small share of weights that matter most, judged by activation sizes, by scaling them before rounding.Source: AWQ: Activation-aware Weight Quantization (Lin et al., 2023)Explained in Quantization: Running LLMs in 8 and 4 Bits

B

Backpropagation: The algorithm that computes the slope of the loss for every weight in one backward pass, applying the chain rule from the loss back through each layer.Source: Learning representations by back-propagating errors (Rumelhart, Hinton & Williams, 1986)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
Bank conflict: When threads of one warp access different addresses in the same one of shared memory's 32 banks, so the hardware serializes the accesses.Source: CUDA Programming Guide: writing CUDA kernels (NVIDIA)Explained in The GPU Memory Hierarchy: Registers, Caches and DRAM, Measured
Base model: A model trained only to predict the next token of internet-scale text, before any instruction tuning. It continues text rather than following instructions.Source: Training language models to follow instructions with human feedback (Ouyang et al., 2022)Explained in How a Large Language Model Works: The Big Picture
Beam search: A decoding method that keeps the k most probable partial outputs at each step and extends each one, instead of committing to a single token.Source: Generation strategies (Hugging Face docs)
BEIR: A benchmark of 18 retrieval datasets from different domains, used to test how well retrievers work outside the data they were trained on.Source: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of IR Models (Thakur et al., 2021)
τ-bench: An agent benchmark in which an LLM plays a customer and the agent follows a written policy while calling tools that change a database. A run passes if the final database matches the goal state.Source: τ-bench (Yao et al., 2024)Explained in Evaluating Agents: pass@k, pass^k and Run-to-Run Noise, Measured
Benchmark contamination: Test questions, or close copies of them, appearing in a model's training data, which inflates its benchmark scores.Source: Language Models are Few-Shot Learners, Appendix C (Brown et al., 2020)Explained in Build an LLM From Scratch: Evaluating a Base Model
BERTBidirectional Encoder Representations from Transformers: Google's 2018 encoder-only Transformer, pretrained to fill in masked words using context on both sides. Widely used for classification and as the base of embedding models.Source: BERT (Devlin et al., 2018)Explained in Encoders, Decoders and GPT, and How Sampling Works
Best-of-N: Generating N candidate answers and keeping the one a verifier or reward model scores highest.Source: Let’s Verify Step by Step (Lightman et al., 2023)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
BF16bfloat16: A 16-bit floating-point format with the same range as 32-bit floats but less precision. The usual format for training and serving model weights.Source: BFloat16: The secret to high performance on Cloud TPUs (Google Cloud, 2019)
Bi-encoder: A model that embeds the query and each document separately, so document vectors can be computed ahead of time and searched fast. The usual first-stage retriever.Source: Sentence-BERT (Reimers & Gurevych, 2019)Explained in Adding a Reranker and Proving It Helped
Bias correction: Adam's division of its moving averages by 1 − βᵗ, which undoes the pull toward zero from starting them at zero.Source: Adam: A Method for Stochastic Optimization (Kingma & Ba, 2014)Explained in Optimizers: From Gradient Descent to AdamW
Binade: The floats between two consecutive powers of two, which are evenly spaced.Source: What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg, 1991)Explained in Floating Point and Quantization: The Math of Fewer Bits
Bits per byte: A model's total loss in bits divided by the number of UTF-8 bytes in the text. Unlike perplexity, it doesn't depend on the tokenizer.Source: The Pile (Gao et al., 2020)Explained in Entropy, Cross-Entropy and Perplexity
BLEU: A score for machine translation that measures how much a translation overlaps with human reference translations, from 0 to 100.Source: BLEU: a Method for Automatic Evaluation of Machine Translation (Papineni et al., 2002)Explained in How a Large Language Model Works: The Big Picture
Block table: A per-request lookup table in a paged KV cache that maps each logical block of a sequence to the physical block in GPU memory that holds it, like an operating system's page table.Source: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)Explained in PagedAttention and vLLM: Paging the KV Cache Like an OS
BM25: A classic keyword-ranking formula. It scores a document by how often the query words appear in it, giving rare words more weight and damping repeats and long documents.Source: The Probabilistic Relevance Framework: BM25 and Beyond (Robertson & Zaragoza, 2009)Explained in BM25 vs Dense vs Hybrid Search: When to Use Each
Bootstrap: Estimating uncertainty by resampling your test results, with replacement, thousands of times and seeing how much the metric moves.Source: Bootstrap Methods: Another Look at the Jackknife (Efron, 1979)
BPEByte-pair encoding: A way to build a tokenizer's vocabulary: start from single characters or bytes and repeatedly merge the most frequent pair of neighboring symbols into a new token until the vocabulary reaches a set size.Source: Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2016)Explained in Tokenization: How LLMs Turn Text into Tokens
Bradley-Terry model: A model of paired comparisons where the chance A beats B is the sigmoid of their score difference. It gives the standard reward-model and DPO losses.Source: Direct Preference Optimization (Rafailov et al., 2023)Explained in From Base Model to Assistant: SFT, RLHF and DPO
BrowseComp: An OpenAI benchmark of 1,266 hard-to-find, easy-to-verify questions for web-browsing agents.Source: BrowseComp (Wei et al., 2025)Explained in Research Agents: Web Search, Fetch and Citations, Measured
Bubblewrap: A small, unprivileged Linux tool that starts a process in new namespaces with only the mounts you list. Claude Code's sandbox uses it on Linux and WSL2.Source: Bubblewrap (GitHub)Explained in Sandboxing and Permissions for AI Agents, Measured
Budget forcing: Controlling how long a model thinks by ending its reasoning early, or extending it by suppressing the end-of-thinking token and appending "Wait".Source: s1: Simple test-time scaling (Muennighoff et al., 2025)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
Bus bandwidth (busbw): nccl-tests' measure of how well a collective uses the links: the algorithm bandwidth (size over time) times a factor such as 2(n−1)/n for all-reduce, comparable to the link's peak.Source: nccl-tests performance notes (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
Byte fallback: A tokenizer rule that spells any character missing from the vocabulary as its UTF-8 bytes, so no input is ever mapped to an unknown token.Source: Llama 2 (Touvron et al., 2023)Explained in Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens

C

Cache breakpoint: A cache_control marker in a Claude API request that sets where the cached prompt prefix ends; everything before it can be reused by later requests.Source: Prompt caching (Claude API docs)Explained in What an AI Agent Is: The Loop, Measured
Cache hit rate: The share of work served from a cache instead of being done again. For prompt caching, the share of input tokens read from the cache at a fraction of the normal input price.Source: Prompt caching (Claude API docs)Explained in Cost and Latency of Agents: Caching, Effort, Models and Batch, Measured
Cache-aware routing: Sending each request to the replica most likely to already hold its prompt prefix in its KV cache.Source: llm-d: Precise prefix-cache aware routingExplained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
CacheBlend: A method that reuses a document's KV cache at a new position in the prompt by recomputing a small subset of tokens to repair cross-attention.Source: CacheBlend (Yao et al., 2024)Explained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
Caching allocator: PyTorch's GPU memory manager. It keeps freed blocks in a pool for reuse instead of calling cudaMalloc and cudaFree, which are slow and can synchronize the device.Source: CUDA semantics (PyTorch 2.11 docs)Explained in Kernel Fusion, torch.compile and CUDA Graphs, Measured
Calibration set: A small sample of typical inputs run through a model to measure activation ranges before quantizing it.Source: GPTQ (Frantar et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
CaMeL: A prompt-injection defense: a privileged LLM plans from the trusted user request only, a quarantined LLM reads untrusted data without tool access, and an interpreter tracks where every value came from and enforces policies.Source: Defeating Prompt Injections by Design (Debenedetti et al., 2025)Explained in Defending Tool-Using Agents Against Prompt Injection by Design
Canary release: Releasing a change to a small slice of traffic first and comparing its metrics with the rest before rolling it out further.Source: Canarying Releases (Google SRE Workbook)Explained in Rolling Back a Bad Prompt Change in Production
Canary token: A unique fake secret or marker planted where only a leak could expose it. If it ever shows up somewhere else, such as a sandbox, a log or a model request, something leaked.Source: Canarytokens documentation (Thinkst)Explained in Sandboxing and Permissions for AI Agents, Measured
Capacity factor: A multiplier on each expert's even share of a batch's tokens. Tokens beyond that limit are dropped and skip the expert through the residual connection.Source: Switch Transformers (Fedus, Zoph & Shazeer, 2021)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
Catastrophic cancellation: The loss of accuracy when two nearly equal rounded numbers are subtracted, leaving mostly their rounding errors.Source: Algorithms for Computing the Sample Variance (Chan, Golub & LeVeque, 1983)Explained in Reductions and Normalization Kernels: Summing Across Threads
Catastrophic forgetting: When fine-tuning on a narrow task makes a model lose general abilities it had before.Source: Overcoming catastrophic forgetting in neural networks (Kirkpatrick et al., 2017)Explained in Fine-Tuning a Small Model with LoRA on 5,000 Examples
Causal attention: Attention in which each token can only look at itself and earlier tokens. It lets a model generate left to right and cache past keys and values.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in The KV Cache: What It Stores, What It Costs, How to Shrink It
CDCChange data capture: Streaming every insert, update and delete from a source system as it happens, so downstream copies such as search indexes stay in sync.Source: Change data capture (Wikipedia)
Central difference: A numerical derivative, (f(x+h) - f(x-h)) / 2h. Its error shrinks with h squared, against h for the one-sided version, until rounding takes over.Source: torch.autograd.gradcheck (PyTorch docs)Explained in Derivatives, Gradients and the Chain Rule
cgroup: Control group: a Linux kernel feature that caps resources such as memory, CPU and the number of processes for a group of processes.Source: Control Group v2 (Linux kernel docs)Explained in Sandboxing and Permissions for AI Agents, Measured
Chain of thought: Intermediate reasoning steps a model writes before its final answer. Each step becomes context for the steps that follow.Source: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
Chain rule of probability: The probability of a sequence equals the product of each item's probability given everything before it. It is why a next-token predictor can score any text.Source: A Neural Probabilistic Language Model (Bengio et al., 2003)Explained in Probability and the Chain Rule of Language
Chat template: The fixed layout of special tokens and role labels a chat model expects a conversation to be wrapped in before tokenization.Source: Meta Llama 3 prompt formatExplained in Tokenization: How LLMs Turn Text into Tokens
Chinchilla: DeepMind's 2022 70B model, trained on 1.4T tokens, which beat the 280B Gopher on the same compute and showed that parameters and data should grow together.Source: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Chunked prefill: Splitting a long prompt's prefill into fixed-size chunks and running one chunk per step alongside other users' decode tokens, so a long prompt doesn't freeze everyone else's stream.Source: SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills (Agrawal et al., 2023)Explained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
Chunking: Splitting documents into passages small enough to embed and retrieve one by one. Where you cut decides what the retriever can find.Source: Chunking Strategies for LLM Applications (Pinecone)Explained in Chunking Legal Contracts for Retrieval
Client toolset: A bundle of Anthropic-defined tools, such as the computer or browser toolset, declared with one tools entry and executed by your own code.Source: Browser use tool (Claude API docs)Explained in Computer-Use and Browser Agents: Pixels Versus the Page
CLIPContrastive Language-Image Pre-training: An OpenAI model with an image encoder and a text encoder, trained on 400 million image-caption pairs so that matching images and captions get similar vectors.Source: Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)Explained in Multimodal Models: How LLMs See and Hear
Cohen’s kappa: An agreement score between two raters that corrects for agreement by chance: 0 is chance level, 1 is perfect. Used to check an LLM judge against humans.Source: A Coefficient of Agreement for Nominal Scales (Cohen, 1960)Explained in Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift
ColBERT: A late-interaction retriever: it keeps one vector per token and scores a document by matching each query token to its best document token. More accurate than one vector per text, at higher storage cost.Source: ColBERT (Khattab & Zaharia, 2020)
Common Crawl: A non-profit, free archive of web crawls going back to 2007, with more than 300 billion pages. Most open pretraining datasets start from it.Source: Common CrawlExplained in Pretraining: Data, Scaling Laws and Training at Scale
Compaction: Replacing the older part of a conversation with a model-written summary so a long task can continue in a smaller context.Source: Compaction (Claude API docs)Explained in Context Engineering: Keeping a Long Agent Run Inside the Window
Complex number: A number a + bi with a real part and an imaginary part, where i² = −1. Pictured as the point (a, b), multiplying two of them multiplies their lengths and adds their angles.Source: RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)Explained in Rotations, Complex Numbers and the Math of RoPE
Computational graph: A record of a calculation as a network of small operations, each node holding a value and pointing to the values it was computed from. Autograd walks it backward to compute gradients.Source: Automatic differentiation in machine learning: a survey (Baydin et al., 2015)Explained in Backpropagation by Hand: From the Chain Rule to Autograd
Compute capability: NVIDIA's version number for a GPU's features and limits: 8.0 for A100, 8.9 for Ada, 9.0 for H100.Source: CUDA Programming Guide: compute capabilities (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Compute-bound: Work whose speed is limited by the chip’s arithmetic throughput because each byte read is used for many calculations. LLM prefill is the classic example.Source: Making Deep Learning Go Brrrr From First Principles (Horace He)Explained in How LLM Inference Works: Prefill, Decode and the Memory Wall
Compute-optimal: The model size and token count that reach the lowest loss for a fixed training compute budget: about 20 tokens per parameter in the Chinchilla study.Source: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)Explained in Pretraining: Data, Scaling Laws and Training at Scale
compute-sanitizer: NVIDIA's runtime checker for out-of-bounds accesses, races and uninitialized reads in CUDA programs.Source: Compute Sanitizer documentation (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Computer use: An agent setup where the model sees a screen through screenshots and controls it with mouse and keyboard actions at pixel coordinates, which your code carries out.Source: Computer use tool (Claude API docs)Explained in Computer-Use and Browser Agents: Pixels Versus the Page
Condition number: The ratio of the largest curvature (or singular value) to the smallest. High values make plain gradient descent zig-zag and converge slowly.Source: Deep Learning, chapter 8 (Goodfellow, Bengio & Courville, 2016)Explained in Optimizers: From Gradient Descent to AdamW
Conditional probability: The probability of one event given that another has happened: p(B | A) = p(A, B) / p(A).Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Probability and the Chain Rule of Language
Confidence interval: A range that likely contains the true value, given sampling noise. If two variants’ intervals overlap a lot, the difference between them may not be real.Source: Adding Error Bars to Evals (Miller, 2024)Explained in Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift
Constrained decoding: Restricting which tokens the model may produce at each step, so the output always matches a grammar or JSON schema.Source: Efficient Guided Generation for Large Language Models (Willard & Louf, 2023)Explained in Guaranteeing Valid JSON Output from LLMs
Context editing: A Claude API feature that clears old tool results or thinking blocks from the prompt, server-side, once it passes a token threshold.Source: Context editing (Claude API docs)Explained in Context Engineering: Keeping a Long Agent Run Inside the Window
Context engineering: Choosing and maintaining what goes into a model's context window over a long task (tool results, history, notes) so it stays accurate and affordable.Source: Effective context engineering for AI agents (Anthropic, 2025)Explained in Context Engineering: Keeping a Long Agent Run Inside the Window
Context rot: The decline in a model's accuracy at recalling or using information as its input grows longer.Source: Context Rot (Chroma, 2025)
Context window: The maximum number of tokens a model can take in at once, prompt and output combined.Source: Context windows (Claude API docs)
Continuous batching: Adding new requests to the running batch at every generation step instead of waiting for the whole batch to finish, which keeps the GPU busy.Source: Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., 2022)Explained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
Contrastive learning: Training that pulls the vectors of matching pairs together and pushes mismatched pairs apart, for example scoring each image against every caption in a batch.Source: Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)Explained in Multimodal Models: How LLMs See and Hear
Control task: A probing baseline that gives every word type one fixed random label, so a probe can only score by memorizing words. Real accuracy minus control accuracy, the selectivity, shows how much the representation did.Source: Designing and Interpreting Probes with Control Tasks (Hewitt & Liang, 2019)Explained in Linear Probes: Reading What a Model's Activations Encode
Copy-on-write: Letting several sequences share one memory block until one of them needs to change it; that sequence then gets its own copy. Used so parallel samples and beam search candidates can share KV cache blocks.Source: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)Explained in PagedAttention and vLLM: Paging the KV Cache Like an OS
Cosine schedule: A learning-rate schedule that lowers the rate along a half cosine curve from its peak to a small final value over a planned number of steps.Source: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Cosine similarity: A measure of how closely two vectors point the same way, from 1 (same direction) through 0 (unrelated) to -1 (opposite): their dot product divided by both lengths.Source: Vector embeddings (OpenAI API docs)Explained in Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
Cost per completed task: Total spend on an agent's runs, failed runs included, divided by the runs that succeeded. Equal to cost per run divided by success rate.Source: AI Agents That Matter (Kapoor et al., 2024)Explained in Cost and Latency of Agents: Caching, Effort, Models and Batch, Measured
CPTCall-performance threshold: The share of calls a router must send to the strong model to recover a given part of the quality gap; CPT(50%) is the share needed for half of it.Source: RouteLLM: Learning to Route LLMs with Preference Data (Ong et al., 2024)Explained in Routing Requests Between a Cheap and a Strong Model
Credit assignment: Working out which of many actions or tokens deserve credit for a reward that arrives only at the end.Source: Group-in-Group Policy Optimization (Feng et al., 2025)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
Critical batch size: The batch size, in tokens per step, beyond which a bigger batch stops reducing the number of steps needed to reach a given loss. It grows as the loss falls during training.Source: Scaling Laws for Neural Language Models (Kaplan et al., 2020)Explained in Build an LLM From Scratch: The Training Loop
Cross-attention: Attention where the queries come from one sequence (the decoder) and the keys and values from another (the encoder's output), so each output token can look at the whole input. It needs no causal mask.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Reading "Attention Is All You Need", the 2017 Transformer Paper
Cross-encoder: A model that reads the query and a passage together and outputs a relevance score. More accurate than comparing separate embeddings, but too slow for the whole corpus, so it is used to rerank a short list.Source: Passage Re-ranking with BERT (Nogueira & Cho, 2019)Explained in Adding a Reranker and Proving It Helped
Cross-entropy loss: The standard training measure for language models: low when the model gave high probability to the token that actually came next, high when it was surprised.Source: Scaling Laws for Neural Language Models (Kaplan et al., 2020)Explained in How a Large Language Model Works: The Big Picture
Crosscoder: A sparse dictionary trained to rebuild activations from several layers or several models at once, so each latent has a direction in each.Source: Sparse Crosscoders for Cross-Layer Features and Model Diffing (Lindsey et al., 2024)Explained in Interpretability for Safety: Lie Detectors, Sycophancy and Audits
CUADContract Understanding Atticus Dataset: About 500 commercial contracts with clauses labeled by lawyers across 41 categories, used to test contract review and retrieval.Source: CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review (Hendrycks et al., 2021)
CUB: NVIDIA's CUDA C++ library of block-wide and device-wide primitives such as reduce, scan and sort, part of CCCL.Source: cub::DeviceReduce (NVIDIA CCCL docs)Explained in Reductions and Normalization Kernels: Summing Across Threads
cuBLAS: NVIDIA's GPU library of linear-algebra routines, which provides the matrix multiplications PyTorch uses. cuBLASLt is its lighter API for matrix multiplies only.Source: cuBLAS documentation (NVIDIA)Explained in From torch.matmul to Silicon: What Happens When PyTorch Runs
CUDA graph: A recorded sequence of GPU operations that the CPU launches with one call, removing per-kernel launch overhead. Shapes and memory addresses must stay fixed.Source: Getting Started with CUDA Graphs (NVIDIA, 2019)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
CUDA stream: A queue of GPU operations, such as kernels and memory copies, that run in the order they were added. Work in different streams may run at the same time.Source: CUDA Programming Guide: asynchronous execution (NVIDIA)Explained in From torch.matmul to Silicon: What Happens When PyTorch Runs
CuTe: CUTLASS 3's C++ layout library, which describes data and thread arrangements as shape:stride layouts.Source: CuTe layouts (NVIDIA CUTLASS docs)Explained in Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
CUTLASS: NVIDIA's open-source CUDA C++ templates for writing fast matrix-multiply kernels on CUDA cores and tensor cores.Source: CUTLASS (NVIDIA, GitHub)Explained in Writing a Fast Matmul in CUDA, Step by Step

D

Dark matter (interpretability): The part of a model's activations that a sparse autoencoder or transcoder fails to reconstruct, left as an unexplained error term.Source: Decomposing The Dark Matter of Sparse Autoencoders (Engels et al., 2024)Explained in Limits and Open Problems in Interpretability
Data exfiltration: Getting private data out of a system without permission, for example by having an agent put secrets into a URL it fetches.Source: Web fetch tool (Claude API docs)Explained in Research Agents: Web Search, Fetch and Citations, Measured
Data parallelism: Running several complete copies of a model and giving each its own requests. In MoE serving, data-parallel attention means each GPU runs attention for its own requests and keeps its own KV cache.Source: vLLM: Data Parallel DeploymentExplained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
DAUDaily active users: The number of distinct users who use a product on a given day.Source: Active users (Wikipedia)
Dead latent: A sparse-autoencoder latent that stops firing during training and never recovers, so its parameters are wasted.Source: Scaling and evaluating sparse autoencoders (Gao et al., 2024)Explained in Sparse Autoencoders: Pulling Features Out of a Model
Decode: The phase after prefill where the model generates one token at a time. Each step re-reads the weights and KV cache, so it is limited by memory bandwidth.Source: Efficiently Scaling Transformer Inference (Pope et al., 2022)Explained in How LLM Inference Works: Prefill, Decode and the Memory Wall
Decoder-only model: A Transformer with a causal mask, trained to predict the next token: the design behind GPT and most modern LLMs.Source: Improving Language Understanding by Generative Pre-Training (Radford et al., 2018)Explained in Encoders, Decoders and GPT, and How Sampling Works
Decoupled RoPE: MLA's way of adding rotary positions: a small extra query and key part carries RoPE, so the rest of the key can stay compressed and its up-projection can be folded into the query.Source: DeepSeek-V2 (DeepSeek-AI, 2024)Explained in The Math of Attention: Scaling, Masks, Jacobians and Low Rank
Deduplication: Removing exact and near-duplicate documents or lines from training data, which improves models and reduces memorization.Source: The FineWeb Datasets (Penedo et al., 2024)Explained in Pretraining: Data, Scaling Laws and Training at Scale
DeepEP: DeepSeek's open-source library of fast all-to-all GPU kernels for dispatching tokens to experts and combining the results in MoE models.Source: DeepEP (DeepSeek)Explained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
DeepGEMM: DeepSeek's open-source library of tensor-core matrix multiplication kernels, first for FP8 with fine-grained scaling and now also FP4 and fused MoE layers.Source: DeepGEMM (DeepSeek)Explained in Case Study: DeepSeek's Inference Stack and DeepInfer
DeepInfer: DeepSeek's production inference engine, open-sourced in October 2026 under Apache 2.0. Derived from vLLM, it serves DeepSeek-V4.1-Flash on NVIDIA Blackwell GPUs and Huawei Ascend NPUs.Source: DeepInfer (DeepSeek)Explained in Case Study: DeepSeek's Inference Stack and DeepInfer
DeepSeek Sparse Attention: Attention in DeepSeek-V3.2 where a small FP8 indexer scores every past token and the main attention reads only the top 2,048.Source: DeepSeek-V3.2-Exp technical report (DeepSeek-AI, 2025)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Delayed and current scaling: Two ways to choose FP8 scales. Current scaling measures amax from the tensor being cast; delayed scaling reuses the amax values seen in earlier steps, which saves a pass but can clip a new outlier.Source: Transformer Engine: Using FP8 and FP4 (NVIDIA)Explained in Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
Dense retrieval: Search that turns the query and documents into embeddings and returns the nearest ones, matching on meaning rather than exact words.Source: Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., 2020)Explained in BM25 vs Dense vs Hybrid Search: When to Use Each
Dequantization: Turning quantized integers back into floating-point values by multiplying by their scale and adding a zero point or minimum. 4-bit kernels do it in registers just before the multiply.Source: MARLIN (Frantar et al., 2024)Explained in Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
Derivative: The slope of a function at a point: how much its output changes per unit change of its input, the value rise over run settles on as the step shrinks to zero.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Derivatives, Gradients and the Chain Rule
Difference of means: A direction computed as the average activation over examples with a property minus the average over examples without it. Used for probing and steering.Source: Refusal in Language Models Is Mediated by a Single Direction (Arditi et al., 2024)Explained in Steering With Activations: The Refusal Direction, Measured
Diffusion model: A generative model that starts from random noise and removes it step by step to produce an image or other data.Source: Denoising Diffusion Probabilistic Models (Ho et al., 2020)Explained in Multimodal Models: How LLMs See and Hear
Direct logit attribution: Splitting a model's output score into what each layer or attention head writes directly into the final residual stream, by projecting each piece onto the answer's unembedding direction. It shows what wrote the answer, not what was needed.Source: A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)Explained in Opening the Black Box: What Interpretability Can Explain
Directional ablation: Removing the component of every activation that lies along one direction, at every layer and position, so the model can no longer represent it.Source: Refusal in Language Models Is Mediated by a Single Direction (Arditi et al., 2024)Explained in Steering With Activations: The Refusal Direction, Measured
Disaggregated serving: Running prefill and decode on separate GPU pools, so each can be sized and tuned for its own bottleneck, and moving the KV cache between them.Source: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., 2024)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
DiskANN: A graph-based ANN index designed to keep most of its data on SSD instead of RAM, so billion-vector indexes fit on one machine.Source: DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node (Subramanya et al., 2019)
Distillation: Training a smaller model to imitate a larger one’s outputs, keeping much of its quality on a task at a fraction of the cost.Source: Distilling the Knowledge in a Neural Network (Hinton et al., 2015)Explained in Build an LLM From Scratch: Distillation and Synthetic Data
Distributional simplicity bias: The finding that neural networks learn low-order statistics of their data, such as token frequencies, before higher-order ones.Source: Neural Networks Learn Statistics of Increasing Complexity (Belrose et al., 2024)Explained in What Training Builds Inside a Model, Snapshot by Snapshot
doctest: Python's convention and standard-library module for examples written as interactive sessions (a >>> line followed by its expected output) inside documentation, which can be run to check they still print that output.Source: doctest (Python docs)Explained in An Agent End to End: A Docs-Maintenance Agent, Measured
Dot product: Multiply two equal-length lists of numbers position by position and add up the results. It is large when the vectors point the same way, and it is how LLMs score attention and logits.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
Double buffering: Keeping two copies of a tile so the next one loads while the current one is being computed on, hiding memory latency.Source: Efficient GEMM in CUDA (NVIDIA CUTLASS docs)Explained in Writing a Fast Matmul in CUDA, Step by Step
DPODirect preference optimization: Training on pairs of preferred and rejected answers so the model favors the preferred kind, without a separate reward model.Source: Direct Preference Optimization (Rafailov et al., 2023)Explained in From Base Model to Assistant: SFT, RLHF and DPO
Dr. GRPO: A variant of GRPO that drops the division by the group's reward standard deviation and by each reply's length, removing biases toward too-easy or too-hard questions and toward long wrong answers.Source: Understanding R1-Zero-Like Training (Liu et al., 2025)Explained in Build an LLM From Scratch: RL with Verifiable Rewards (GRPO)
Draft model: A small, fast model, or extra layers on the main model, that guesses the next few tokens so a larger target model can check them all in one pass during speculative decoding.Source: Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Dropout: Randomly zeroing a fraction of a network's values during training so it can't rely on any single one. It is turned off when the model is used.Source: Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava et al., 2014)
DSpark: DeepSeek-V4.1's speculative decoding: a built-in drafter proposes several tokens and a confidence head decides how many of them to verify.Source: DeepSeek-V4.1-Flash model cardExplained in Case Study: DeepSeek's Inference Stack and DeepInfer
Dual-batch overlap: Splitting a batch into two micro-batches so one computes while the other's expert-parallel communication is in flight, hiding network time.Source: DeepSeek-V3/R1 Inference System Overview (DeepSeek, 2025)Explained in Case Study: DeepSeek's Inference Stack and DeepInfer
Dynamic filtering: A mode of Claude's web search and fetch tools in which the model writes code that filters results before they enter its context window.Source: Web search tool (Claude API docs)Explained in Research Agents: Web Search, Fetch and Citations, Measured

E

Eager mode: PyTorch's default way of running a model: each operation launches as its own GPU kernel as Python reaches it, with no whole-graph compilation or fusion.Source: Introduction to torch.compile (PyTorch)Explained in Build an LLM From Scratch: Writing the Model
EAGLE: A family of speculative decoding methods whose lightweight draft layer reads the target model's internal features to propose tokens.Source: EAGLE (Li et al., 2024)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Early fusion: Training one model on mixed text and image (or audio) tokens from the start of pretraining, instead of attaching a separately trained encoder later.Source: Chameleon (Chameleon Team, 2024)Explained in Multimodal Models: How LLMs See and Hear
ECEExpected calibration error: How far a model’s stated confidence is from its actual accuracy, averaged over confidence buckets. 0 means 70%-confident predictions are right 70% of the time.Source: On Calibration of Modern Neural Networks (Guo et al., 2017)
Echo Trap: A failure of multi-turn agent RL in which rollouts lose diversity and the model repeats one rewarded template until training collapses.Source: RAGEN (Wang et al., 2025)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
Effective context length: The longest input at which a model still performs above a quality threshold, often well below its advertised context window.Source: RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., 2024)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Effort level: A Claude API setting, from low to max, that steers how much a model writes, including how often and how long it thinks. It is a behavioral signal, not a strict token budget.Source: Effort (Claude API docs)Explained in Reasoning and Acting: ReAct, Planning and Thinking, Measured
Einsum: A notation that names each axis of each array with a letter. Letters that appear in the inputs but not in the output are multiplied and summed over.Source: torch.einsum (PyTorch docs)Explained in The Math You Need for LLMs, and the Shapes Inside One
Elicitation: An MCP client feature that lets a server ask the user for input through a form, or send them to a URL for sensitive steps.Source: MCP specification: ElicitationExplained in MCP: Connecting Agents to Tools
Embedding: A list of numbers (a vector) that a model produces for a piece of text, arranged so that texts with similar meaning end up close together.Source: Sentence-BERT (Reimers & Gurevych, 2019)Explained in BM25 vs Dense vs Hybrid Search: When to Use Each
Emergent abilities: Abilities absent in smaller models that appear in larger ones. How many are real, rather than an effect of all-or-nothing scoring, is debated.Source: Emergent Abilities of Large Language Models (Wei et al., 2022)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Encoder-decoder: A model in two parts: an encoder reads the whole input into one vector per token, and a decoder generates the output one token at a time while attending to the encoder's vectors. The original 2017 Transformer and T5 use it.Source: Sequence to Sequence Learning with Neural Networks (Sutskever et al., 2014)Explained in Reading "Attention Is All You Need", the 2017 Transformer Paper
Encoder-only model: A Transformer with no causal mask, so every token attends to every other. It produces vectors for understanding tasks instead of generating text.Source: BERT (Devlin et al., 2018)Explained in Encoders, Decoders and GPT, and How Sampling Works
Endpoint Picker: The component a gateway consults for every inference request. It filters and scores model-server pods on cache contents, load and latency targets and returns the best one.Source: llm-d: Endpoint PickerExplained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
Entropy: The average surprise of a random source, minus the sum of p log p. It is the fewest bits per symbol any lossless code can average.Source: A Mathematical Theory of Communication (Shannon, 1948)Explained in Entropy, Cross-Entropy and Perplexity
EPLBExpert Parallelism Load Balancer: An algorithm that decides which MoE experts to duplicate and where to place the copies, so every GPU gets a similar share of expert work.Source: EPLB (DeepSeek)Explained in Case Study: DeepSeek's Inference Stack and DeepInfer
Error node: A node in an attribution graph holding the part of an MLP's output the transcoder fails to reproduce, so the graph still adds up to the real model.Source: Circuit Tracing (Ameisen et al., 2025)Explained in Circuits and Attribution Graphs, Traced on a Laptop GPU
Euler's formula: e^(iθ) = cos θ + i sin θ: the complex number of length 1 at angle θ, so multiplying by it rotates a point by θ.Source: RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)Explained in Rotations, Complex Numbers and the Math of RoPE
Eval set: A fixed collection of test cases that every change is run against, so quality is measured instead of guessed.Source: Define success criteria and build evaluations (Claude API docs)Explained in Building an Eval Suite for a Customer Support Bot
Evaluation awareness: A model recognizing that it is probably being tested, which can make its good behavior in evaluations unrepresentative.Source: Claude Sonnet 4.5 System Card (Anthropic, 2025)Explained in Interpretability for Safety: Lie Detectors, Sycophancy and Audits
Evaluator-optimizer: A workflow where one LLM call generates a response and another evaluates it and gives feedback, in a loop, until the result meets the criteria.Source: Building effective agents (Anthropic)Explained in Reflection and Self-Correction in Agents, Measured
Expandable segments: A PyTorch allocator option that grows one virtually contiguous segment by mapping physical pages on demand, which reduces fragmentation.Source: CUDA semantics (PyTorch 2.11 docs)Explained in Kernel Fusion, torch.compile and CUDA Graphs, Measured
Expected value: The probability-weighted average of a random quantity's possible values. The average of many samples approaches it.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Probability and the Chain Rule of Language
Expert parallelism: Placing a mixture-of-experts model's experts on different GPUs and sending each token only to the GPUs holding its chosen experts, using all-to-all dispatch and combine.Source: DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)Explained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
Exponential backoff: Waiting longer after each failed retry, usually doubling the wait up to a cap, so a struggling service gets time to recover.Source: Timeouts, retries and backoff with jitter (AWS Builders’ Library)Explained in Long-Horizon Reliability: Compounding Errors, Retries and Checkpoints
Exponential moving average: A running average where each new value gets weight 1 − β and older values shrink by β per step, so it averages over about 1/(1 − β) steps.Source: Adam: A Method for Stochastic Optimization (Kingma & Ba, 2014)Explained in Optimizers: From Gradient Descent to AdamW

F

F1 score: The harmonic mean of precision and recall: a single number that is high only when both are high.Source: F-score (Wikipedia)
Fail-to-pass test: A test that fails before a bug fix and passes after it. Benchmarks use these to check that the fix works.Source: SWE-bench (Jimenez et al., 2023)Explained in Coding Agents: How They Fix Real Bugs, Measured
Faithfulness: Whether every claim in an answer is supported by the retrieved context, whether or not it happens to be true in the world.Source: RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)Explained in Detecting and Reducing LLM Hallucinations
Feature absorption: When a more specific latent takes over part of a broader feature, so the broad latent silently misses some cases.Source: A is for Absorption (Chanin et al., 2024)Explained in Sparse Autoencoders: Pulling Features Out of a Model
Feature dimensionality: The fraction of a hidden dimension a feature gets: 1 means a dimension of its own, 1/2 an antipodal pair, 0 a dropped feature.Source: Toy Models of Superposition (Elhage et al., 2022)Explained in Superposition: How Models Store More Features Than Neurons
Feature flag: A switch in configuration that turns a change on or off for some or all users without a new deploy, which makes rollbacks instant.Source: Feature Toggles (aka Feature Flags) (Hodgson, martinfowler.com, 2017)Explained in Rolling Back a Bad Prompt Change in Production
Feature splitting: As a sparse autoencoder's dictionary grows, one latent's concept divides into several narrower latents.Source: Towards Monosemanticity (Bricken et al., 2023)Explained in Sparse Autoencoders: Pulling Features Out of a Model
Few-shot prompting: Putting a few solved examples of a task in the prompt before the real question, so the model can infer the task without training. Zero-shot uses none.Source: Language Models are Few-Shot Learners (Brown et al., 2020)Explained in Build an LLM From Scratch: Evaluating a Base Model
FineWeb: Hugging Face's open 15-trillion-token English web dataset built from 96 Common Crawl snapshots. FineWeb-Edu is its 1.3-trillion-token educational subset.Source: The FineWeb Datasets (Penedo et al., 2024)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Flash-Decoding: A decode-time attention method that splits the KV cache into chunks processed in parallel, then combines the partial results using each chunk's log-sum-exp.Source: Flash-Decoding for long-context inference (Dao et al., 2023)Explained in Writing an Attention Kernel: FlashAttention and Decode in Triton
FlashAttention: An exact attention algorithm that works in tiles held in fast on-chip memory, so it never writes the full n×n score matrix to the GPU's main memory.Source: FlashAttention (Dao et al., 2022)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
FlashInfer: An attention kernel library for LLM serving that handles paged and ragged KV caches as block-sparse matrices.Source: FlashInfer (Ye et al., 2025)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
FlashMLA: DeepSeek's open-source attention kernels for multi-head latent attention, including sparse attention over a paged KV cache.Source: FlashMLA (DeepSeek)Explained in Case Study: DeepSeek's Inference Stack and DeepInfer
FLOPsFloating-point operations: A count of arithmetic work, such as the operations needed to process one token. Compare TFLOPS, a hardware speed per second.Source: Transformer Inference Arithmetic (kipply, 2022)
FLORES-200: A benchmark of about 2,000 Wikipedia sentences professionally translated into 200 languages, used to compare models and tokenizers across languages.Source: No Language Left Behind (NLLB Team, 2022)Explained in Tokenization: How LLMs Turn Text into Tokens
FP16: IEEE half precision: 5 exponent and 10 mantissa bits. More precise than BF16, but it overflows above 65,504.Source: FP8 Formats for Deep Learning (Micikevicius et al., 2022)Explained in Floating Point and Quantization: The Math of Fewer Bits
FP4 (NVFP4, MXFP4): 4-bit floating-point formats. NVFP4 and MXFP4 add a shared scale for each small block of values to keep accuracy; Blackwell-class GPUs run them natively.Source: Introducing NVFP4 for Efficient and Accurate Low-Precision Inference (NVIDIA, 2025)Explained in Quantization: Running LLMs in 8 and 4 Bits
FP8: An 8-bit floating-point format. It halves memory compared with BF16 and runs faster on recent GPUs, usually with little quality loss.Source: FP8 Formats for Deep Learning (Micikevicius et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
Frobenius norm: The size of a matrix measured as if it were one long vector: the square root of the sum of all its entries squared.Source: The Approximation of One Matrix by Another of Lower Rank (Eckart & Young, 1936)Explained in Low Rank and the SVD: How Much of a Matrix Really Matters
Full jitter: Picking each retry wait at random between zero and the current backoff, so many clients that failed together don't all retry at the same moment.Source: Timeouts, retries and backoff with jitter (AWS Builders’ Library)Explained in Long-Horizon Reliability: Compounding Errors, Retries and Checkpoints
Fused multiply-add (FMA): One instruction that computes a × b + c with a single rounding, counted as two floating-point operations. A matrix multiply is almost entirely FMAs.Source: NVIDIA Ada GPU Architecture whitepaper (NVIDIA)Explained in Writing a Fast Matmul in CUDA, Step by Step
Fused optimizer: An optimizer implementation that updates every parameter tensor, its weights and moment estimates, in one GPU kernel pass instead of a chain of separate elementwise kernels.Source: PyTorch AdamW documentationExplained in Build an LLM From Scratch: Making Training Fast on One GPU

G

GAIA: A benchmark of 466 questions for general AI assistants, each with one short answer, in three difficulty levels. They are easy for people (92%) and were hard for models at release.Source: GAIA (Mialon et al., 2023)Explained in Evaluating Agents: pass@k, pass^k and Run-to-Run Noise, Measured
GaLore: A training method that projects each weight matrix's gradient onto its top singular directions, refreshed periodically, so optimizer state stays small while every weight still trains.Source: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (Zhao et al., 2024)Explained in Low Rank and the SVD: How Much of a Matrix Really Matters
Gated DeltaNet: A gated linear-attention layer that updates a fixed-size state with a decay gate and the delta rule. Hybrid models such as Qwen3-Next use it in most of their layers.Source: Gated Delta Networks: Improving Mamba2 with Delta Rule (Yang et al., 2024)Explained in The Math of Attention: Scaling, Masks, Jacobians and Low Rank
GDPRGeneral Data Protection Regulation: The EU law on how personal data may be collected, stored, processed and deleted, including people’s right to have their data erased.Source: Regulation (EU) 2016/679, the General Data Protection Regulation
GELUGaussian error linear unit: A smooth activation function, x·Φ(x), where Φ is the standard normal cumulative distribution. Used in GPT-style models.Source: Gaussian Error Linear Units (GELUs) (Hendrycks & Gimpel, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
GEMM / GEMV: General matrix-matrix multiply and general matrix-vector multiply. Decoding at batch size 1 is mostly GEMVs, which are limited by memory bandwidth.Source: cuBLAS documentation (NVIDIA)Explained in From torch.matmul to Silicon: What Happens When PyTorch Runs
Gemma Scope: Google DeepMind's open set of sparse autoencoders and transcoders for every layer of Gemma 2 2B and 9B.Source: Gemma Scope (Lieberum et al., 2024)Explained in Sparse Autoencoders: Pulling Features Out of a Model
Generation stall: A pause in every user's token stream while the GPU processes another request's long prompt in one piece.Source: Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (Agrawal et al., 2024)Explained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
GGUF: The single-file model format used by llama.cpp. It stores quantized weights together with metadata such as the architecture, tokenizer and chat template.Source: GGUF specification (ggml docs)Explained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
Gibbs' inequality: Cross-entropy is never below entropy, H(p, q) ≥ H(p), with equality only when q = p. Equivalently, KL divergence is never negative.Source: Elements of Information Theory (Cover & Thomas)Explained in Entropy, Cross-Entropy and Perplexity
Glitch token: A token that is in a model's vocabulary but almost never appeared in its training data, so its embedding is barely trained and inputs containing it can make the model behave erratically.Source: Fishing for Magikarp (Land and Bartolo, 2024)Explained in Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens
GLUGated linear unit: A layer that computes two projections of its input and multiplies them position by position, with one passed through an activation that acts as a gate. SwiGLU and GEGLU are variants.Source: Language Modeling with Gated Convolutional Networks (Dauphin et al., 2016)Explained in The Modern LLM Block: What Changed Since 2017
Gluon: A lower-level language on Triton's compiler stack that exposes layouts, shared memory and asynchrony to the programmer.Source: Introduction to Gluon (Triton tutorials source)Explained in Triton: Writing GPU Kernels in Python, Measured
Golden set: A fixed set of test inputs with trusted, human-checked answers or labels, used to score every change the same way.Source: Your AI Product Needs Evals (Husain, 2024)Explained in Building an Eval Suite for a Customer Support Bot
Goodput: Throughput that counts only requests that met their latency targets. A server can have high throughput and low goodput.Source: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., 2024)Explained in Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization
GPTQ: A post-training method that quantizes weights to 3 or 4 bits layer by layer, using a small calibration set to compensate for the rounding errors.Source: GPTQ (Frantar et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
GPUDirect RDMA: An NVIDIA technology that lets a network card or other PCIe device read and write GPU memory directly, so data between servers skips CPU memory.Source: GPUDirect RDMA (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
GQAGrouped-query attention: Attention in which several query heads share one key/value head, shrinking the KV cache several times over.Source: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., 2023)Explained in The KV Cache: What It Stores, What It Costs, How to Shrink It
Gradient: The vector of all partial derivatives of a function, one per input or weight. It points in the direction of steepest increase, so training steps against it.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Derivatives, Gradients and the Chain Rule
Gradient accumulation: Running several micro-batches forward and backward and adding up their gradients before one optimizer update, so a large batch can be trained in less memory.Source: Reproducing GPT-2 (124M) in llm.c (Karpathy, 2024)Explained in Build an LLM From Scratch: The Training Loop
Gradient clipping: Capping the size of each training update: if the gradient's overall length exceeds a limit, often 1.0, it is scaled down to that limit.Source: DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Gradient descent: An optimization method that repeatedly moves each parameter a small step against the slope of the loss, to lower it.Source: Deep Learning, Chapter 8 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
Gradient noise scale: The batch size beyond which larger batches stop cutting the number of training steps proportionally.Source: An Empirical Model of Large-Batch Training (McCandlish et al., 2018)Explained in Optimizers: From Gradient Descent to AdamW
Graph break: A point where torch.compile can't capture the code (for example a .item() call), so it splits the program into separately compiled graphs with Python running in between.Source: PyTorch 2.x overview (PyTorch)Explained in Kernel Fusion, torch.compile and CUDA Graphs, Measured
Greedy decoding: Always picking the single most likely next token instead of sampling. Predictable, but prone to flat or repetitive text.Source: Generation strategies (Hugging Face docs)Explained in How a Large Language Model Works: The Big Picture
Grid-stride loop: A kernel loop in which each thread steps through the data by the total number of threads in the grid, so any grid size covers any array.Source: CUDA Pro Tip: Grid-Stride Loops (Harris, 2013)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Grounding: Tying an answer to specific provided sources, so each claim can be traced to a passage and checked against it.Source: Reduce hallucinations (Claude API docs)Explained in Detecting and Reducing LLM Hallucinations
Grouped GEMM: Many independent matrix multiplications that share their widths but differ in row count, run in one kernel launch. MoE layers use it to run all their experts at once.Source: DeepGEMM (DeepSeek)Explained in Mixture-of-Experts Kernels: Routing, Grouped GEMM and Load Imbalance
Grouped ordering: Launching output tiles in small 2D groups rather than row by row, so tiles that run at the same time reuse the same input data in L2.Source: Triton tutorials (Triton docs)Explained in Triton: Writing GPU Kernels in Python, Measured
GRPOGroup relative policy optimization: A reinforcement learning algorithm that samples a group of answers per prompt and scores each against the group's mean and standard deviation, removing PPO's value model.Source: DeepSeekMath (Shao et al., 2024)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
Guardrail: An automatic check around a model or a change, such as a quality metric that must not drop or a filter on inputs and outputs, that blocks or rolls back when it trips.Source: OWASP Top 10 for LLM Applications 2025

H

H2OHeavy-Hitter Oracle: Heavy-Hitter Oracle: a KV cache eviction policy that keeps the tokens with the highest accumulated attention plus the most recent ones.Source: H2O (Zhang et al., 2023)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Hallucination: When a model states something fluent and confident that is false or not supported by its sources.Source: Survey of Hallucination in Natural Language Generation (Ji et al., 2022)Explained in Detecting and Reducing LLM Hallucinations
HBMHigh-bandwidth memory: The stacked memory on a GPU package. Its size limits how much model and KV cache fit; its bandwidth limits how fast decode runs.Source: High Bandwidth Memory (Wikipedia)
He initialization: Starting weights with variance 2/fan-in, twice Xavier's for equal fan-in and fan-out, to make up for ReLU zeroing half its inputs.Source: Delving Deep into Rectifiers (He et al., 2015)Explained in Keeping Signals Stable: Initialization, Normalization and Residuals
Headless browser: A browser that runs without a visible window, driven by code (for example Playwright) for testing or agents.Source: Playwright for PythonExplained in Computer-Use and Browser Agents: Pixels Versus the Page
HellaSwag: A four-option benchmark asking for the most plausible ending to a video caption or WikiHow passage. Its wrong endings were machine-written and kept only if they fooled a classifier.Source: HellaSwag (Zellers et al., 2019)Explained in Build an LLM From Scratch: Evaluating a Base Model
Hessian: The matrix of all second derivatives of a loss. Its eigenvalues are the loss's curvatures in different directions.Source: Deep Learning, chapter 4 (Goodfellow, Bengio & Courville, 2016)Explained in Optimizers: From Gradient Descent to AdamW
HIP: AMD's C++ runtime API and kernel language for its GPUs, closely modeled on CUDA; the HIPIFY tools translate CUDA source to it.Source: HIP documentation (AMD)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
HNSWHierarchical Navigable Small World: A graph-based ANN index: each vector is linked to its neighbors in layered graphs that a search walks greedily. Fast and accurate, but the whole graph must sit in RAM.Source: Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs (Malkov & Yashunin, 2016)Explained in Designing a RAG System for 10 Million Documents
Hybrid search: Running keyword search (BM25) and vector search side by side and fusing their results, so you catch both exact terms and paraphrases.Source: Contextual Retrieval (Anthropic, 2024)Explained in BM25 vs Dense vs Hybrid Search: When to Use Each
HyDEHypothetical Document Embeddings: Asking an LLM to write a plausible answer to the query, then searching with that answer’s embedding, which often resembles the real documents more than the question does.Source: Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao et al., 2022)
Hydra effect: When one part of a model is ablated, later layers change their output to partly make up for it, so the ablation underrates how much the part did.Source: The Hydra Effect (McGrath et al., 2023)Explained in Opening the Black Box: What Interpretability Can Explain

I

ICI: Inter-chip interconnect: the direct links between neighboring TPU chips, arranged as a 2D or 3D torus with no switch inside a slice.Source: TPU7x (Google Cloud)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Idempotency: Repeating a request has the same effect as sending it once. An idempotency key lets the server recognize a retry and avoid doing the work twice.Source: The Idempotency-Key HTTP Header Field (IETF draft)Explained in Streaming LLM Responses over SSE and Surviving Disconnects
Identity matrix: The square matrix with 1s on the diagonal and 0s elsewhere, which leaves every vector unchanged.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Matrices as Transformations, and Why Neural Networks Need a Bend
Implicit reward: In DPO, beta times the log-ratio of the trained model's and the reference model's probability of a reply. DPO trains it to rank the chosen reply above the rejected one, with no separate reward model.Source: Direct Preference Optimization (Rafailov et al., 2023)Explained in Build an LLM From Scratch: Preference Tuning with DPO
Importance matrix: Statistics gathered by running sample text through a model, which llama.cpp's quantizer uses to protect the weights that matter most.Source: llama.cpp quantize READMEExplained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
Importance ratio: The ratio of the new policy's probability to the old policy's for a sample, used to reweight samples drawn from the old policy.Source: Proximal Policy Optimization Algorithms (Schulman et al., 2017)Explained in The Math of Alignment: Policy Gradients, PPO, DPO and GRPO
In-context learning: A model performing a new task from instructions or examples in its prompt, with no weight updates.Source: Language Models are Few-Shot Learners (Brown et al., 2020)Explained in Encoders, Decoders and GPT, and How Sampling Works
In-context learning score: Olsson et al.'s measure of how much a model uses its context: the loss on the 500th token minus the loss on the 50th. More negative means more in-context learning.Source: In-context Learning and Induction Heads (Olsson et al., 2022)Explained in What Training Builds Inside a Model, Snapshot by Snapshot
Induction head: An attention head that completes repeated patterns: after seeing [A][B] earlier, at the next [A] it looks back at what followed and predicts [B]. Linked to learning from the prompt.Source: In-context Learning and Induction Heads (Olsson et al., 2022)Explained in Attention Heads and Induction Heads, Measured in GPT-2
InferencePool: A Kubernetes custom resource from the Gateway API Inference Extension that groups model-server pods and points a gateway at the endpoint picker that routes among them.Source: Gateway API Inference Extension (Kubernetes SIGs)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
InfiniBand: A low-latency network for clusters with remote direct memory access (RDMA). An NDR port runs at 400 Gb/s, 50 GB/s in each direction.Source: InfiniBand Adapters (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
Instruction tuning: Fine-tuning a pretrained model on many tasks phrased as natural-language instructions, so it follows new instructions it wasn't trained on.Source: Finetuned Language Models Are Zero-Shot Learners (Wei et al., 2021)Explained in From Base Model to Assistant: SFT, RLHF and DPO
INT4: 4-bit integer weights, usually with one scale per small group of weights. A quarter of BF16’s memory; common for weight-only quantization such as GPTQ and AWQ.Source: Integer Quantization for Deep Learning Inference (Wu et al., 2020)Explained in Quantization: Running LLMs in 8 and 4 Bits
Interleaved thinking: Model reasoning in thinking blocks between tool calls: after reading one tool result and before choosing the next action.Source: Thinking (Claude API docs)Explained in Reasoning and Acting: ReAct, Planning and Thinking, Measured
Interpretability: Research that studies what a trained model computes by examining its internal activations and weights, not only its outputs. Mechanistic interpretability tries to reverse-engineer that computation into features and circuits.Source: Open Problems in Mechanistic Interpretability (Sharkey et al., 2025)Explained in Inside the Black Box: Features, Superposition and Circuits
Interpretability illusion: A result that seems to locate a mechanism but is produced by the method itself, such as a subspace patch that works by switching on a pathway the model doesn't normally use.Source: Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching (Makelov et al., 2023)Explained in Limits and Open Problems in Interpretability
Intrinsic dimension: The number of free directions a fine-tuning update really needs. It is often tiny compared with the model's parameter count, which is why LoRA works.Source: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (Aghajanyan et al., 2020)Explained in Low Rank and the SVD: How Much of a Matrix Really Matters
Intrinsic self-correction: A model revising its own answer with no outside feedback, so it must judge by itself whether the answer was right.Source: Large Language Models Cannot Self-Correct Reasoning Yet (Huang et al., 2023)Explained in Reflection and Self-Correction in Agents, Measured
Inverse transform sampling: Drawing from a distribution by picking a uniform random number and finding where it falls in the running totals of the probabilities (the CDF).Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Probability and the Chain Rule of Language
Inverted index: A map from each word to the documents that contain it. It is the data structure behind keyword search engines such as BM25.Source: Introduction to Information Retrieval: building an inverted index (Manning et al.)
Irreducible loss: The floor E in a fitted scaling law L = E + A/N^α: the loss no model size can beat, roughly the randomness left in the text.Source: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)Explained in Build an LLM From Scratch: Scaling Laws on a Laptop
IsoFLOP profile: The losses of several model sizes trained on the same compute, with fewer tokens for bigger models. The lowest point marks the compute-optimal size for that budget.Source: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)Explained in Build an LLM From Scratch: Scaling Laws on a Laptop
Iteration-level scheduling: Making the batching decision before every model step instead of once per batch, so finished requests leave and new ones join immediately. The idea behind continuous batching.Source: Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., 2022)Explained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
ITLInter-token latency: The average time between consecutive output tokens while an answer streams. Another name for TPOT.Source: LLM inference benchmarking metrics (NVIDIA NIM docs)Explained in How LLM Inference Works: Prefill, Decode and the Memory Wall
IVF-PQInverted file with product quantization: An ANN index that groups vectors into clusters (IVF) and compresses them into short codes (PQ), so a search scans only a few clusters of small codes. Far less memory, slightly lower recall.Source: Faiss: Guidelines to choose an index (Meta)

J

Jaccard similarity: The size of the intersection of two sets divided by the size of their union: 1 for identical sets, 0 for sets with nothing in common. Near-duplicate detection applies it to documents' word shingles.Source: Mining of Massive Datasets, Chapter 3 (Leskovec, Rajaraman and Ullman)Explained in The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop
Jacobian: The matrix of all partial derivatives of a vector-valued function: one row per output, one column per input. A linear layer's Jacobian is its weight matrix.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Derivatives, Gradients and the Chain Rule
Johnson-Lindenstrauss lemma: A result that many points in a high-dimensional space can be mapped into far fewer dimensions while nearly keeping their distances. It is why high-dimensional spaces hold exponentially many nearly perpendicular directions.Source: Toy Models of Superposition (Elhage et al., 2022)Explained in Vectors and Dot Products: Length, Angle and Meaning
JSON Schema: A standard vocabulary for describing the shape of JSON data: field names, types, allowed values and which fields are required. Tool definitions use it to describe their inputs.Source: JSON SchemaExplained in How Tool Calling Works Inside the Model
JSON-RPC: A small standard for remote procedure calls written as JSON: requests with a method, params and id, responses with a result or error, and notifications that get no reply. MCP messages use version 2.0.Source: JSON-RPC 2.0 SpecificationExplained in MCP: Connecting Agents to Tools
JumpReLU: An activation that passes a value only if it exceeds a learned per-latent threshold. Gemma Scope's SAEs and transcoders use it.Source: Jumping Ahead: JumpReLU Sparse Autoencoders (Rajamanoharan et al., 2024)Explained in Sparse Autoencoders: Pulling Features Out of a Model

K

K-composition: When a later attention head's key matrix reads a subspace an earlier head wrote to, so the earlier head shapes where the later one attends. Induction heads use it to read a previous-token head.Source: A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)Explained in Attention Heads and Induction Heads, Measured in GPT-2
K-quants: llama.cpp's family of 2- to 6-bit quantization types that store weights in super-blocks of smaller blocks, each with its own quantized scale. Q4_K_M averages about 4.89 bits per weight.Source: llama.cpp k-quants pull request #1684Explained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
k3 estimator: A one-sample estimate of KL divergence, r - 1 - log r with r the ratio of reference to policy probability, that is never negative and averages to the true KL; GRPO uses it as its KL penalty.Source: Approximating KL Divergence (John Schulman)Explained in The Math of Alignment: Policy Gradients, PPO, DPO and GRPO
Kernel fusion: Combining several GPU operations into one kernel so intermediate results stay in on-chip memory instead of making round trips to HBM.Source: Making Deep Learning Go Brrrr From First Principles (Horace He)Explained in How GPUs Work for LLMs: Cores, Memory and Number Formats
Kernel launch: Starting a GPU function (a kernel) as many parallel threads. The CPU queues it and returns immediately, and each launch costs microseconds of CPU and driver time.Source: CUDA Programming Guide: programming model (NVIDIA)Explained in From torch.matmul to Silicon: What Happens When PyTorch Runs
KIVI: A 2-bit KV cache quantization method that quantizes keys per channel and values per token and keeps the most recent tokens at full precision.Source: KIVI (Liu et al., 2024)Explained in Quantization: Running LLMs in 8 and 4 Bits
KL penalty: A term that subtracts β times the log-ratio between the tuned model's and the reference model's probabilities, so RLHF can't drift far from the starting model.Source: Learning to summarize from human feedback (Stiennon et al., 2020)Explained in From Base Model to Assistant: SFT, RLHF and DPO
kNNk-nearest neighbors: Exact search for the k vectors closest to a query, comparing it against all of them. Too slow at scale, but it is the ground truth that ANN recall is measured against.Source: Nearest neighbor search (Wikipedia)
KV cache: The attention keys and values already computed for earlier tokens, kept in GPU memory so each new token does not recompute them. Often the main limit on how many requests fit at once.Source: Efficiently Scaling Transformer Inference (Pope et al., 2022)Explained in The KV Cache: What It Stores, What It Costs, How to Shrink It
KV cache eviction: Dropping some cached keys and values during inference, by recency or attention score, to fit long contexts in less memory at some risk to recall.Source: H2O (Zhang et al., 2023)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
KV cache offloading: Moving KV cache blocks that don't fit in GPU memory to CPU memory, local SSD or remote storage, so they can be loaded back later instead of recomputed.Source: KV Offloading Usage Guide (vLLM docs)Explained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
KV transfer: Moving a request's KV cache from the GPU that ran prefill to the GPU that will decode it, usually over RDMA and often layer by layer while prefill is still running.Source: Splitwise (Patel et al., 2023)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode

L

L2 cache: The GPU's last on-chip cache, shared by all SMs. All traffic to DRAM goes through it: 32 MB on an RTX 4060 Laptop GPU, 50 MB on an H100.Source: Nsight Compute Kernel Profiling Guide (NVIDIA)Explained in The GPU Memory Hierarchy: Registers, Caches and DRAM, Measured
Label smoothing: A training trick that gives the correct token slightly less than 100% of the target probability and spreads the rest over all other tokens, so the model learns to be less overconfident.Source: Rethinking the Inception Architecture for Computer Vision (Szegedy et al., 2015)Explained in Reading "Attention Is All You Need", the 2017 Transformer Paper
Last-Event-ID: The header a browser sends when an SSE connection reconnects, carrying the ID of the last event it received, so the server can resume from there.Source: HTML Standard: Server-sent events (WHATWG)Explained in Streaming LLM Responses over SSE and Surviving Disconnects
Late interaction: Encoding query and document separately into per-token vectors, then comparing them token by token at search time. It sits between bi-encoders and cross-encoders in cost and accuracy.Source: ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (Santhanam et al., 2021)
Layer normalization: Rescales one token's vector so its numbers have mean 0 and spread 1, then applies a learned gain and bias. It uses no statistics from other examples in the batch.Source: Layer Normalization (Ba, Kiros & Hinton, 2016)Explained in Multi-Head Attention and the Transformer Block
LC win rateLength-controlled win rate: AlpacaEval’s win rate adjusted to remove the judge’s preference for longer answers.Source: Length-Controlled AlpacaEval (Dubois et al., 2024)
ldmatrix: A warp instruction that loads one to four 8×8 matrices of 16-bit values from shared memory straight into MMA fragment registers.Source: PTX ISA (NVIDIA)Explained in Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
LDS: Local data share: the programmer-managed on-chip memory in each AMD compute unit, AMD's equivalent of CUDA shared memory (64 KB on CDNA 3, 160 KB on CDNA 4).Source: AMD CDNA 4 architecture whitepaperExplained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Learning rate: The step-size setting that scales each training update. Too small trains slowly; too large makes training oscillate or diverge.Source: Deep Learning, Chapter 8 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
Learning-rate warm-up: Starting training with a tiny learning rate and raising it over the first few thousand steps before letting it decay, which keeps early updates from destabilizing the model.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Reading "Attention Is All You Need", the 2017 Transformer Paper
Least squares: Fitting a line or curve by choosing the parameters that minimize the sum of squared gaps between predictions and data points.Source: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)Explained in Counting: Parameters, FLOPs, Memory and Scaling Laws
Length-normalized accuracy: Multiple-choice accuracy where each option's log-likelihood is divided by its length in characters before the best is picked, so short options aren't favored just for being short.Source: Multiple Choice Normalization in LM Evaluation (Gao, 2021)Explained in Build an LLM From Scratch: Evaluating a Base Model
Likelihood displacement: DPO-style training lowering the probability of the preferred replies too, so probability moves to other replies, sometimes unwanted ones.Source: Unintentional Unalignment (Razin et al., 2024)Explained in Build an LLM From Scratch: Preference Tuning with DPO
Linear attention: A variant of attention that replaces the softmax of query-key scores with a dot product of transformed vectors, so the product can be reordered to cost time linear in sequence length and run as a recurrence with a fixed-size state.Source: Transformers are RNNs (Katharopoulos et al., 2020)Explained in The Math of Attention: Scaling, Masks, Jacobians and Low Rank
Linear map: A function on vectors that keeps sums and scaling: mapping x + y gives the sum of the mapped x and y, and mapping c times x gives c times the mapped x. Every matrix is one.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Matrices as Transformations, and Why Neural Networks Need a Bend
Linear probe: A small linear classifier trained on a layer's activations to test whether some property can be read from them.Source: Understanding intermediate layers using linear classifier probes (Alain & Bengio, 2016)Explained in Linear Probes: Reading What a Model's Activations Encode
llama.cpp: An open-source C/C++ engine for running LLMs locally on CPUs and GPUs (Metal, CUDA, Vulkan and others). LM Studio and Ollama use it to run GGUF models.Source: llama.cpp (ggml-org)Explained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
LLM-as-a-judge: Using an LLM to grade outputs against a rubric or to pick the better of two answers. Cheap and scalable, but biased, so it must be checked against human labels.Source: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)Explained in Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift
llm-d: An open-source, Kubernetes-native stack for distributed LLM inference. It runs vLLM or SGLang underneath and adds cache-aware routing, prefill/decode disaggregation and autoscaling. A CNCF sandbox project.Source: llm-d (GitHub)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
LLM.int8(): An 8-bit inference method that multiplies most values in INT8 but keeps the few outlier feature dimensions in 16 bits, avoiding the accuracy loss that appears in models around 6.7B parameters and up.Source: LLM.int8() (Dettmers et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
LM Studio: A desktop app for running local LLMs through llama.cpp, or MLX on Apple silicon, with a model browser and an OpenAI-compatible server.Source: LM Studio docsExplained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
lm-evaluation-harness: EleutherAI's open-source framework for running standard benchmarks on language models with fixed prompts and scoring rules.Source: lm-evaluation-harness (EleutherAI)Explained in Build an LLM From Scratch: Evaluating a Base Model
LMCache: An open-source KV cache layer that stores caches outside GPU memory (CPU, disk, remote stores) and shares them across vLLM and SGLang instances.Source: LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference (Liu et al., 2025)Explained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
Load-balancing loss: An extra training loss that is lowest when a mixture-of-experts router spreads tokens evenly across experts, so a few experts don't take all the work.Source: Switch Transformers (Fedus, Zoph & Shazeer, 2021)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
Locality-sensitive hashing: Hashing that puts similar items in the same bucket with high probability, so likely matches are found without comparing every pair. With MinHash, each signature is split into bands and documents that agree on any whole band become candidates.Source: Mining of Massive Datasets, Chapter 3 (Leskovec, Rajaraman and Ullman)Explained in The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop
Log-derivative trick: The identity ∇p = p ∇log p, which turns the gradient of an expected value into an average that can be estimated from samples.Source: Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (Williams, 1992)Explained in The Math of Alignment: Policy Gradients, PPO, DPO and GRPO
Log-log plot: A chart with both axes on logarithmic scales, where each step is a factor of 10, so power laws appear as straight lines.Source: Scaling Laws for Neural Language Models (Kaplan et al., 2020)Explained in Counting: Parameters, FLOPs, Memory and Scaling Laws
Log-probability: The natural log of a probability, always 0 or negative. Models add log-probabilities instead of multiplying probabilities, which would round to zero.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Probability and the Chain Rule of Language
Log-softmax: Log-probabilities computed directly from logits as z minus log-sum-exp(z), instead of taking the log of softmax, which hits log 0 when tiny probabilities underflow.Source: torch.nn.functional.log_softmax (PyTorch docs)Explained in Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely
Log-sum-exp: The log of the sum of exponentials of a list of scores. It lies between the largest score and that score plus ln n, so it acts as a smooth maximum; a token's log-probability is its logit minus the LSE.Source: FlashAttention-2 (Dao, 2023)Explained in Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely
Logit difference: The logit of the correct answer minus the logit of a competing answer. Nearly linear in the residual stream, it ignores anything that boosts both answers equally.Source: How to use and interpret activation patching (Heimersheim & Nanda, 2024)Explained in Activation Patching and Causal Tracing, Measured in GPT-2
Logit lens: Applying a model's final layer norm and unembedding to an intermediate layer's vector, to see what it would predict if it stopped there.Source: interpreting GPT: the logit lens (nostalgebraist, 2020)Explained in The Residual Stream and the Logit Lens
Logit soft-capping: Squashing attention scores or output logits with cap × tanh(x / cap), so they stay between −cap and +cap.Source: Gemma 2: Improving Open Language Models at a Practical Size (Gemma Team, 2024)Explained in Keeping Signals Stable: Initialization, Normalization and Residuals
Logits: The raw scores a model outputs for every token in its vocabulary before they are turned into probabilities. Higher means more likely to come next.Source: Transformers, the tech behind LLMs (3Blue1Brown, 2024)Explained in How a Large Language Model Works: The Big Picture
LoRALow-rank adaptation: Fine-tuning that freezes the model and trains small low-rank matrices added to some layers, usually well under 1% of the parameters. Cheap to train, store and swap.Source: LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)Explained in Fine-Tuning a Small Model with LoRA on 5,000 Examples
Loss function: A formula that turns a model's output and the correct answer into one number measuring how wrong the output is. Training tries to make it small.Source: Deep Learning, Chapter 8 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
Loss masking: Leaving some tokens out of the training loss, such as the prompt or tool output in an agent trajectory, so the model learns only from text it should produce.Source: Search-R1 (Jin et al., 2025)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
Loss scaling: Multiplying the loss by a constant before the backward pass so small FP16 gradients don't underflow, then dividing the gradients by it before the update.Source: Mixed Precision Training (Micikevicius et al., 2017)Explained in Floating Point and Quantization: The Math of Fewer Bits
Loss spike: A sudden rise in training loss caused by an update that destabilizes the model, so the loss stays high or diverges, unlike batch-to-batch noise. Large runs recover by restarting from an earlier checkpoint and skipping the batches involved.Source: PaLM: Scaling Language Modeling with Pathways (Chowdhery et al., 2022)Explained in Build an LLM From Scratch: The Pretraining Run
Lost in the middle: The finding that LLMs use information at the start and end of a long prompt better than information buried in the middle.Source: Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
LPU: Language processing unit: Groq's deterministic, compiler-scheduled inference chip, which keeps weights in on-chip SRAM instead of HBM.Source: Think Fast: A Tensor Streaming Processor (Abts et al., 2020)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators

M

M-RoPEMultimodal RoPE: A RoPE variant that gives each token three position numbers (time, height and width), so image and video tokens keep their layout.Source: Qwen2-VL (Wang et al., 2024)Explained in Multimodal Models: How LLMs See and Hear
Machete: vLLM's mixed-input matrix-multiply kernel for Hopper GPUs (4- or 8-bit weights, 16-bit activations), built on CUTLASS and wgmma as Marlin's successor.Source: Introducing Machete (Red Hat, 2024)Explained in Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
Magpie: A way to generate instruction data: an aligned chat model is given only its template's user-turn prefix, so it writes a question, and is then asked to answer it.Source: Magpie (Xu et al., 2024)Explained in Build an LLM From Scratch: Supervised Fine-Tuning
Mantissa: The bits of a floating-point number that hold its significant digits. Their count sets the format's relative precision.Source: What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg, 1991)Explained in Floating Point and Quantization: The Math of Fewer Bits
Marlin: A GPU kernel that multiplies 16-bit activations by 4-bit weights, staying near the ideal speedup over FP16 up to batch sizes of about 16 to 32. vLLM uses it.Source: MARLIN (Frantar et al., 2024)Explained in Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
Masked language modeling: A pretraining task that hides some input tokens (15% in BERT) and trains the model to predict them from the surrounding text.Source: BERT (Devlin et al., 2018)Explained in Encoders, Decoders and GPT, and How Sampling Works
Mass-mean probe: A probe direction found without training: the average activation of true examples minus the average of false ones.Source: The Geometry of Truth (Marks & Tegmark, 2023)Explained in Linear Probes: Reading What a Model's Activations Encode
Master weights: An FP32 copy of the weights that the optimizer updates during mixed-precision training, so updates too small for 16-bit formats are kept.Source: Mixed Precision Training (Micikevicius et al., 2017)Explained in Floating Point and Quantization: The Math of Fewer Bits
Matrix multiplication: Combining an m×k and a k×n matrix into an m×n matrix, where each entry is the dot product of a row of the first with a column of the second. It costs about 2mkn FLOPs.Source: numpy.matmul (NumPy docs)Explained in The Math You Need for LLMs, and the Shapes Inside One
Matryoshka embeddings: Embeddings trained so the first dimensions carry the most information. You can cut vectors short (say from 1024 to 256 numbers) to save storage at a modest quality cost.Source: Matryoshka Representation Learning (Kusupati et al., 2022)
Max trick: Subtracting the largest score from every score before computing softmax. The result is unchanged, but no exponential can overflow.Source: Deep Learning, chapter 4 (Goodfellow, Bengio & Courville)Explained in Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely
MCPModel Context Protocol: An open protocol for connecting models to tools and data sources through a standard server interface.Source: Model Context ProtocolExplained in MCP: Connecting Agents to Tools
Mean ablation: Removing a component's contribution by replacing its output with its average over a set of inputs (or, in zero ablation, with zeros), then measuring how the model's output changes.Source: Interpretability in the Wild (Wang et al., 2022)Explained in Opening the Black Box: What Interpretability Can Explain
Medusa: A speculative decoding method that adds extra prediction heads to a model's last layer to guess several future tokens, verified together with tree attention.Source: Medusa (Cai et al., 2024)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Memory coalescing: The GPU combining the addresses of a warp's 32 threads into as few 32-byte memory transactions as possible. Consecutive threads reading consecutive words need the fewest.Source: CUDA Programming Guide: writing CUDA kernels (NVIDIA)Explained in The GPU Memory Hierarchy: Registers, Caches and DRAM, Measured
Memory fragmentation: Memory that is allocated or free but unusable. Internal fragmentation is unused space inside an allocation; external fragmentation is free space split into gaps too small for the next request.Source: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)Explained in PagedAttention and vLLM: Paging the KV Cache Like an OS
Memory snapshot: A PyTorch record of every GPU allocation and free, with the Python stack that made it, used to see what occupies memory at a training step's peak.Source: Understanding CUDA Memory Usage (PyTorch)Explained in Build an LLM From Scratch: Fitting Bigger Models in Memory
Memory tool: A client-side Claude tool that lets the model create, read and edit files in a /memories directory your code stores, so notes survive a context reset.Source: Memory tool (Claude API docs)Explained in Context Engineering: Keeping a Long Agent Run Inside the Window
Memory-bound: Work whose speed is limited by how fast data can be read from memory, not by how fast the chip can calculate. LLM decode is the classic example.Source: Making Deep Learning Go Brrrr From First Principles (Horace He)Explained in How LLM Inference Works: Prefill, Decode and the Memory Wall
Message Batches API: A Claude API mode that takes many requests at once, processes them asynchronously (most batches finish within an hour, all within 24 hours) and bills them at half the normal price.Source: Batch processing (Claude API docs)Explained in Cost and Latency of Agents: Caching, Effort, Models and Batch, Measured
MFMA: Matrix fused multiply-add: AMD's matrix-core instructions, run by a whole wavefront together, the counterpart of NVIDIA's mma instructions.Source: Matrix cores on CDNA (AMD ROCm blog)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
MFUModel FLOPs utilization: The share of a GPU’s peak compute that the model actually uses. Low MFU means the GPU spends most of its time waiting on memory, communication or scheduling.Source: PaLM: Scaling Language Modeling with Pathways (Chowdhery et al., 2022)Explained in Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization
MHAMulti-head attention: Multi-head attention: attention split into several heads that each have their own queries, keys and values, as in the original Transformer. It has the largest KV cache of the common designs.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in The KV Cache: What It Stores, What It Costs, How to Shrink It
Micro-batch: The group of sequences processed in one forward and backward pass; several micro-batches are accumulated into one optimizer step.Source: Reproducing GPT-2 (124M) in llm.c (Karpathy, 2024)Explained in Build an LLM From Scratch: What a Laptop Can Train
Microscaling (MX): OCP block formats in which every 32 values share one 8-bit power-of-two (E8M0) scale. Blackwell tensor cores apply the scales in hardware.Source: OCP Microscaling Formats (MX) v1.0 SpecificationExplained in Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
microVM: A stripped-down virtual machine with its own kernel that boots in around 100 ms, used to isolate untrusted code more strongly than a container.Source: Firecracker specificationExplained in Sandboxing and Permissions for AI Agents, Measured
Mid-training: A late pretraining stage that upsamples the highest-quality web text, curated sources and synthetic data, typically a few percent of the total training compute.Source: 2 OLMo 2 Furious (OLMo team, 2024)Explained in Build an LLM From Scratch: From Our Model to Frontier Models
Min-p sampling: Keeping only tokens whose probability is at least min_p times the top token's, then sampling among them.Source: Turning Up the Heat: Min-p Sampling (Nguyen et al., 2024)Explained in Encoders, Decoders and GPT, and How Sampling Works
MinHash: A hashing technique that gives each document a short signature, so near-duplicate documents can be found without comparing every pair.Source: The FineWeb Datasets (Penedo et al., 2024)Explained in The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop
Mini-batch: A random sample of training examples used to estimate the gradient for one optimizer step. Its noise shrinks like one over the square root of its size.Source: Deep Learning, chapter 8 (Goodfellow, Bengio & Courville, 2016)Explained in Optimizers: From Gradient Descent to AdamW
Mixed-precision training: Training with 16-bit (or 8-bit) matrix math while keeping sums and master weights in FP32.Source: Mixed Precision Training (Micikevicius et al., 2017)Explained in Floating Point and Quantization: The Math of Fewer Bits
Mixtral: Mistral AI's open-weight mixture-of-experts models. Mixtral 8x7B has 8 experts per layer, uses 2 per token, and has 46.7B total and 12.9B active parameters.Source: Mixtral of Experts (Jiang et al., 2024)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
MLAMulti-head latent attention: DeepSeek’s attention design that caches one small compressed vector per token instead of full keys and values, shrinking the KV cache further than GQA.Source: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (DeepSeek-AI, 2024)Explained in The KV Cache: What It Stores, What It Costs, How to Shrink It
MLPMulti-layer perceptron: The feed-forward network inside each Transformer block. It processes every token’s vector on its own, after attention has mixed in context.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in How a Large Language Model Works: The Big Picture
MLX: Apple's array framework for machine learning on Apple silicon, where arrays live in memory shared by the CPU and GPU. Its mlx-lm package runs and fine-tunes LLMs.Source: MLX (Apple)Explained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
MMA instruction: A tensor-core instruction one warp issues to compute D = A×B + C on small matrix tiles, such as 16×8×16 for 16-bit inputs, with operands spread across the 32 threads' registers.Source: PTX ISA (NVIDIA)Explained in Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
MoBAMixture of Block Attention: Mixture of Block Attention: each query scores blocks of context by their average key and attends only to the top few, like experts in a mixture of experts.Source: MoBA: Mixture of Block Attention for Long-Context LLMs (Lu et al., 2025)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Model bandwidth utilization (MBU): The model's bytes times tokens per second, divided by the GPU's peak memory bandwidth: how close batch-1 decode comes to the memory limit.Source: Accelerating Generative AI with PyTorch II: GPT, Fast (PyTorch)Explained in One Token, Kernel by Kernel: A Decode Step on a Laptop GPU
Model cascade: Trying a cheap model first and escalating to a stronger one only when the first answer looks unreliable.Source: FrugalGPT (Chen, Zaharia & Zou, 2023)Explained in Routing Requests Between a Cheap and a Strong Model
Model collapse: The degradation seen when models are trained repeatedly on earlier models' outputs instead of real data: rare parts of the original distribution disappear.Source: AI models collapse when trained on recursively generated data (Shumailov et al., 2024)Explained in Build an LLM From Scratch: Distillation and Synthetic Data
Model diffing: Comparing a fine-tuned model's internals with those of the model it came from, to find what the extra training added or changed.Source: Sparse Crosscoders for Cross-Layer Features and Model Diffing (Lindsey et al., 2024)Explained in Interpretability for Safety: Lie Detectors, Sycophancy and Audits
Model merging: Combining several fine-tuned checkpoints of the same model into one, usually by averaging or interpolating their weights.Source: SmolLM3 (Hugging Face, 2025)Explained in Build an LLM From Scratch: From Our Model to Frontier Models
Model organism: A model built or studied in depth so its mechanisms or behaviors are known, used as a test bed for interpretability or safety methods, by analogy with lab animals.Source: Open Problems in Mechanistic Interpretability (Sharkey et al., 2025)Explained in Limits and Open Problems in Interpretability
MoEMixture of experts: A model whose layers hold many “expert” sub-networks, of which only a few run for each token. Large in total size, but with the compute per token of a much smaller model.Source: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
Momentum: An optimizer technique that steps along a moving average of recent gradients, so directions that keep flipping cancel and steady ones build up.Source: Deep Learning, chapter 8 (Goodfellow, Bengio & Courville, 2016)Explained in Optimizers: From Gradient Descent to AdamW
Monosemantic: Describes a neuron or feature that responds to one interpretable property of the input.Source: Towards Monosemanticity (Bricken et al., 2023)Explained in Superposition: How Models Store More Features Than Neurons
Mooncake: Moonshot AI's serving platform for Kimi, built around a cluster-wide KV cache pool in CPU memory and SSD, with separate prefill and decode machines.Source: Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (Qin et al., 2024)Explained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
MQAMulti-query attention: Multi-query attention: every attention head has its own query, but all heads share one key and value head, which shrinks the KV cache by the number of heads at some cost in quality.Source: Fast Transformer Decoding: One Write-Head is All You Need (Shazeer, 2019)Explained in The KV Cache: What It Stores, What It Costs, How to Shrink It
MRRMean reciprocal rank: The average of 1/rank of the first relevant result across queries. 1 means the right answer is always first; 0.5 means it is typically second.Source: Mean reciprocal rank (Wikipedia)Explained in Adding a Reranker and Proving It Helped
MT-Bench: A set of 80 multi-turn questions graded by a strong LLM judge, from the paper that popularized LLM-as-a-judge.Source: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
MTEBMassive Text Embedding Benchmark: A public leaderboard comparing embedding models across many tasks and languages.Source: MTEB: Massive Text Embedding Benchmark (Muennighoff et al., 2022)
Multi-token prediction: Training a model with extra heads or modules that predict tokens beyond the next one. The extra predictions can serve as drafts for speculative decoding, as in DeepSeek-V3.Source: Better & Faster Large Language Models via Multi-token Prediction (Gloeckle et al., 2024)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Multimodal model: A model that takes in more than one kind of data, such as text plus images or audio, usually by turning each into vectors in one shared sequence.Source: Flamingo (Alayrac et al., 2022)Explained in Multimodal Models: How LLMs See and Hear
Muon: An optimizer for weight matrices that replaces each momentum update with an orthogonalized version, all singular values near 1, computed by Newton-Schulz iterations.Source: Muon is Scalable for LLM Training (Liu et al., 2025)Explained in Optimizers: From Gradient Descent to AdamW
MXU: Matrix multiply unit: the systolic array inside each TPU TensorCore, 128 × 128 cells before TPU v6e and 256 × 256 from v6e on.Source: Cloud TPU system architecture (Google)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators

N

N-gram model: A model that predicts the next token only from counts of the few tokens before it. A bigram model uses just the current token.Source: Speech and Language Processing, Chapter 3: N-gram Language Models (Jurafsky & Martin)Explained in Probability and the Chain Rule of Language
Name mover head: An attention head in GPT-2 small's indirect-object circuit that attends to a name earlier in the sentence and copies it into the prediction.Source: Interpretability in the Wild (Wang et al., 2022)Explained in Activation Patching and Causal Tracing, Measured in GPT-2
Nat: A unit of information that uses the natural log. 1 nat is about 1.443 bits; LLM training losses are reported in nats.Source: A Mathematical Theory of Communication (Shannon, 1948)Explained in Entropy, Cross-Entropy and Perplexity
NCCLNVIDIA Collective Communications Library: NVIDIA's library of GPU-to-GPU communication operations, such as all-reduce and all-to-all, over NVLink, PCIe, InfiniBand or Ethernet.Source: NCCL User Guide (NVIDIA)Explained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
nDCGNormalized discounted cumulative gain: A ranking score from 0 to 1 that rewards putting the most relevant results nearest the top. 1 means the ideal order.Source: Cumulated Gain-Based Evaluation of IR Techniques (Järvelin & Kekäläinen, 2002)Explained in Adding a Reranker and Proving It Helped
Needle in a haystack: A long-context test that hides one fact in filler text and asks the model to retrieve it.Source: RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., 2024)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
NERNamed entity recognition: A model that finds and labels names of people, organizations, places and similar entities in text. Used to catch PII that fixed patterns miss.Source: Named-entity recognition (Wikipedia)Explained in Building a PII Redaction Gateway in Front of Your LLM
Neural audio codec: A neural network that compresses audio into a stream of integer codes and decodes them back into sound. The codes can serve as audio tokens for a language model.Source: High Fidelity Neural Audio Compression (Défossez et al., 2022)Explained in Multimodal Models: How LLMs See and Hear
Neural Engine: Apple's dedicated on-chip AI block, separate from the GPU, which apps reach through Core ML.Source: Apple introduces M4 chip (Apple)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Neuron: One unit of a neural network: it multiplies its inputs by learned weights, adds a bias, and passes the total through an activation function.Source: Deep Learning, Chapter 6 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
NeuronCore: The compute core of AWS Trainium chips: a systolic Tensor Engine plus vector, scalar and GPSIMD engines sharing a software-managed SRAM.Source: NeuronCore-v3 (AWS Neuron docs)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
NF44-bit NormalFloat: A 4-bit format whose levels are spaced to suit normally distributed weights. QLoRA uses it to store the frozen base model.Source: QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)Explained in Fine-Tuning a Small Model with LoRA on 5,000 Examples
NIXLNVIDIA Inference Xfer Library: An open-source library for fast point-to-point data transfer between machines in inference systems, used to move KV caches between prefill and decode GPUs over RDMA.Source: NIXL (NVIDIA)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
NKI: Neuron Kernel Interface: AWS's Python-based, tile-level kernel language for Trainium, similar in style to Triton.Source: NKI documentation (AWS)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
NLINatural language inference: Classifying whether one text entails, contradicts or is neutral to another. Used to check whether an answer’s claims follow from its sources.Source: A large annotated corpus for learning natural language inference (Bowman et al., 2015)Explained in Detecting and Reducing LLM Hallucinations
Non-embedding parameters: A model's parameter count without its token-embedding table. Kaplan et al.'s scaling laws use it; Chinchilla counts all parameters.Source: Scaling Laws for Neural Language Models (Kaplan et al., 2020)Explained in Build an LLM From Scratch: Scaling Laws on a Laptop
NoPE: No positional encoding: a decoder Transformer trained without any explicit position signal, which can still infer position from its causal mask.Source: The Impact of Positional Encoding on Length Generalization in Transformers (Kazemnejad et al., 2023)Explained in Positional Encoding: How Transformers Know Word Order
Norm (of a vector): The length of a vector: the square root of the sum of its squared entries, which is Pythagoras in any number of dimensions.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Vectors and Dot Products: Length, Angle and Meaning
NSANative Sparse Attention: DeepSeek's trainable sparse attention that combines compressed token summaries, selected token blocks and a sliding window, blended by a learned gate.Source: Native Sparse Attention (Yuan et al., 2025)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Nsight Compute: NVIDIA's kernel profiler (ncu). It replays one GPU kernel many times to collect hardware counters: throughput against peak, memory traffic, occupancy and warp stall reasons.Source: Nsight Compute Kernel Profiling Guide (NVIDIA)Explained in Profiling and the Roofline: Finding What Limits a GPU Kernel
Nsight Systems: NVIDIA's system-wide profiler (nsys). It records a timeline of CPU threads, CUDA API calls, NVTX ranges and GPU work across a whole program.Source: Nsight Systems User Guide (NVIDIA)Explained in Profiling and the Roofline: Finding What Limits a GPU Kernel
NTK-aware scaling: Stretching a RoPE model's context window by raising the RoPE base, which slows the slow rotation pairs a lot and the fast ones barely at all.Source: YaRN: Efficient Context Window Extension of Large Language Models (Peng et al., 2023)Explained in Rotations, Complex Numbers and the Math of RoPE
Nucleus sampling: Sampling from the smallest set of most likely tokens whose probabilities add up to at least p. Also called top-p sampling.Source: The Curious Case of Neural Text Degeneration (Holtzman et al., 2019)Explained in Encoders, Decoders and GPT, and How Sampling Works
NVIDIA Dynamo: NVIDIA's open-source orchestration layer above vLLM, SGLang and TensorRT-LLM, with disaggregated serving, KV-aware routing, KV cache offloading and latency-driven autoscaling.Source: NVIDIA Dynamo (GitHub)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
NVLink: NVIDIA’s high-bandwidth connection between GPUs in a server or rack, much faster than PCIe. Tensor parallelism depends on it.Source: NVLink and NVLink Switch (NVIDIA)
NVLink SHARP: In-network reduction: NVSwitch chips add up the data GPUs send them and return the total, so an all-reduce needs fewer steps and fewer GPU cores.Source: NVIDIA Hopper Architecture In-Depth (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
NVSHMEM: NVIDIA's library for GPU-initiated communication: GPU threads read and write buffers in other GPUs' symmetric memory without a transfer launched by the CPU.Source: NVSHMEM (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
NVSwitch: A switch chip that connects every GPU in a server, or a rack, to every other at full NVLink speed.Source: NVIDIA Hopper Architecture In-Depth (NVIDIA, 2022)Explained in How GPUs Work for LLMs: Cores, Memory and Number Formats

O

Occupancy: Active warps on an SM divided by the most it supports (48 on compute capability 8.9). Too little leaves memory latency unhidden.Source: CUDA C++ Best Practices Guide (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
OCROptical character recognition: Turning images of text, such as scanned pages or photos, into machine-readable text.Source: Optical character recognition (Wikipedia)
Off-policy data: Training data written by a model other than the one being trained, such as preference pairs whose replies came from larger models. On-policy data is sampled from the model itself.Source: Tülu 3 (Lambert et al., 2024)Explained in Build an LLM From Scratch: Preference Tuning with DPO
Ollama: A tool that downloads, stores and serves open-weight models locally with one command, with its own API and an OpenAI-compatible one.Source: Ollama docsExplained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
On-policy distillation: Training a student model on its own sampled outputs to match a teacher's token probabilities. DeepSeek-V4 used it to merge specialist models into one.Source: DeepSeek-V4 (DeepSeek-AI, 2026)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
One-hot vector: A vector with a 1 in one position and 0 everywhere else, used to represent a single token. Multiplying it by the embedding table picks out that token's row.Source: Using the Output Embedding to Improve Language Models (Press & Wolf, 2016)Explained in Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
Online softmax: Computing softmax in one streaming pass by keeping a running maximum and running sum, rescaling earlier totals when a larger value appears. The result is exact.Source: Online normalizer calculation for softmax (Milakov & Gimelshein, 2018)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
OpenTelemetry: An open standard and set of SDKs for emitting traces, metrics and logs that any compatible backend can collect.Source: What is OpenTelemetry? (OpenTelemetry docs)Explained in Tracing Every Step of a Multi-Agent Workflow
Optimizer offload: Keeping optimizer state and the weight update in CPU memory while the GPU runs the forward and backward passes, trading PCIe copies and CPU time for GPU memory.Source: ZeRO-Offload (Ren et al., 2021)Explained in Build an LLM From Scratch: Fitting Bigger Models in Memory
Orchestrator-workers: An agent design in which a lead model breaks a task into subtasks, hands each to a worker model, and combines their results.Source: Building effective agents (Anthropic, 2024)Explained in Multi-Agent Systems: Orchestrators, Sub-agents and A2A, Measured
Orthogonal matrix: A matrix whose transpose is its inverse. It preserves lengths and angles, as rotations and reflections do.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong, 2020)Explained in Rotations, Complex Numbers and the Math of RoPE
OSWorld: A benchmark of 369 tasks on a real desktop virtual machine, each checked by a script that inspects the final state. OSWorld-Verified is the 2025 revision with fixed checkers.Source: OSWorld (Xie et al., 2024)Explained in Evaluating Agents: pass@k, pass^k and Run-to-Run Noise, Measured
Outer product: A column times a row, giving a grid whose entry (i, j) is uᵢ times vⱼ. Every such grid has rank 1.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong, 2020)Explained in Low Rank and the SVD: How Much of a Matrix Really Matters
Outlier features: A few activation dimensions whose values are many times larger than the rest. They make low-bit quantization hard because one large value forces a coarse scale on everything that shares it.Source: LLM.int8() (Dettmers et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
OV circuit: The product of one attention head's value and output matrices: the low-rank map from a token the head attends to into what the head writes back to the residual stream.Source: A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)Explained in The Math of Attention: Scaling, Masks, Jacobians and Low Rank
Overfitting: When a model does much better on its training data than on new data, because it memorized examples instead of learning the general pattern.Source: Deep Learning, Chapter 5 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation

P

μP: A way of scaling initialization and per-layer learning rates with model width so hyperparameters tuned on a small model transfer to a large one.Source: Tensor Programs V (Yang et al., 2022)Explained in Keeping Signals Stable: Initialization, Normalization and Residuals
p95 / p99Percentile latency: p95 is the value that 95% of requests beat, so 1 in 20 is slower; p99 is 1 in 100. Tail percentiles show the slow cases that averages hide.Source: The Tail at Scale (Dean & Barroso, 2013)
Pageable memory: Ordinary host memory that the operating system may move. GPU copies from it go through a staging buffer, and a blocking PyTorch copy from it waits for queued GPU work to finish.Source: A guide on good usage of non_blocking and pin_memory() in PyTorchExplained in Build an LLM From Scratch: Making Training Fast on One GPU
PagedAttention: vLLM’s technique of storing the KV cache in fixed-size blocks, like pages of virtual memory, so little memory is wasted and shared prefixes can reuse blocks.Source: Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023)Explained in PagedAttention and vLLM: Paging the KV Cache Like an OS
Pairwise summation: Adding numbers in pairs, then pairs of pairs, so rounding error grows far more slowly than with a single running total.Source: numpy.sum (NumPy docs)Explained in Floating Point and Quantization: The Math of Fewer Bits
Parallel layers: A Transformer block variant in which attention and the MLP read the same normalized input and both add to the residual stream at once, instead of one after the other.Source: PaLM (Chowdhery et al., 2022)Explained in The Modern LLM Block: What Changed Since 2017
Parallel reduction: Combining many values into one (a sum or a maximum) by pairing them up in a tree, so many threads work at once.Source: Optimizing Parallel Reduction in CUDA (Mark Harris, NVIDIA)Explained in Reductions and Normalization Kernels: Summing Across Threads
Parallel sampling: Generating several independent outputs for the same prompt in one request, for example several candidate answers to pick from.Source: Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)Explained in PagedAttention and vLLM: Paging the KV Cache Like an OS
Parallel tool calls: Several tool calls returned by the model in one response; your code can run them at once and sends all the results back in a single message.Source: Parallel tool use (Claude API docs)Explained in How Tool Calling Works Inside the Model
Partial derivative: The slope of a function of several inputs with respect to one of them, with the others held fixed.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Derivatives, Gradients and the Chain Rule
Partial rollout: In RL training, pausing unfinished long trajectories at the end of an iteration and resuming them in the next, so a few slow runs don't stall the whole batch.Source: Kimi K2 (Kimi Team, 2025)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
Partition function: The normalizing constant Z that makes a reweighted distribution sum to 1. In RLHF it sums over every possible answer, and it cancels in DPO.Source: Direct Preference Optimization (Rafailov et al., 2023)Explained in The Math of Alignment: Policy Gradients, PPO, DPO and GRPO
Pass-to-pass test: A test that passes both before and after a fix. It checks that the fix broke nothing else.Source: SWE-bench (Jimenez et al., 2023)Explained in Coding Agents: How They Fix Real Bugs, Measured
pass@k: The chance that at least one of k attempts at a task succeeds. Estimated from n runs with c successes as 1 − C(n−c, k)/C(n, k), then averaged over tasks.Source: Evaluating Large Language Models Trained on Code (Chen et al., 2021)Explained in Evaluating Agents: pass@k, pass^k and Run-to-Run Noise, Measured
pass^k: The chance that all k attempts at a task succeed, estimated as C(c, k)/C(n, k) from c successes in n runs and averaged over tasks. It measures consistency rather than best-case ability.Source: τ-bench (Yao et al., 2024)Explained in Evaluating Agents: pass@k, pass^k and Run-to-Run Noise, Measured
Patch embedding: The vector made from one small square of an image by flattening its pixels and multiplying by a learned matrix: the image equivalent of a token embedding.Source: An Image is Worth 16x16 Words (Dosovitskiy et al., 2020)Explained in Multimodal Models: How LLMs See and Hear
Path patching: A form of activation patching that changes one component's output only along its connection to chosen later components, so the experiment measures that single path's effect.Source: Interpretability in the Wild (Wang et al., 2022)Explained in Activation Patching and Causal Tracing, Measured in GPT-2
PEFTParameter-efficient fine-tuning: Fine-tuning methods, LoRA being the best known, that train a small number of added or selected parameters instead of the whole model.Source: PEFT documentation (Hugging Face)Explained in Fine-Tuning a Small Model with LoRA on 5,000 Examples
Per-group quantization: Giving each run of consecutive weights (often 128) its own scale, so an outlier only hurts its own group. A 16-bit scale per 128 weights adds 0.125 bits per weight.Source: GPTQ (Frantar et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
Perplexity: e raised to the average next-token loss: roughly how many tokens a model is torn between at each step. Lower is better.Source: Perplexity of fixed-length models (Hugging Face docs)
PIIPersonally identifiable information: Data that can identify a person, such as names, email addresses, phone numbers, and ID or account numbers.Source: NIST SP 800-122: Guide to Protecting the Confidentiality of PII (2010)Explained in Building a PII Redaction Gateway in Front of Your LLM
Pinned memory: CPU memory the operating system may not move, so the GPU's copy engine can read it directly. Host-to-GPU copies need it to run asynchronously and overlap with other work.Source: CUDA Programming Guide: Asynchronous Execution (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
Pipeline parallelism: Splitting a model's layers into stages on different GPUs, so each GPU holds a range of layers and passes activations to the next. It lets big models span servers and raises throughput, but not single-request speed.Source: GPipe (Huang et al., 2018)Explained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
Plan-and-execute: An agent design in which a planner writes the whole sequence of steps before any tool runs and an executor carries it out, asking for a new plan when results don't fit.Source: ReWOO (Xu et al., 2023)Explained in Reasoning and Acting: ReAct, Planning and Thinking, Measured
Policy gradient: The gradient of a model's expected reward, estimated by sampling outputs and weighting the gradient of each output's log-probability by its reward or advantage.Source: Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (Williams, 1992)Explained in The Math of Alignment: Policy Gradients, PPO, DPO and GRPO
Polysemantic: Describes a neuron that responds to several unrelated things, such as citations, HTTP requests and Korean text. Common in language models.Source: Towards Monosemanticity (Bricken et al., 2023)Explained in Superposition: How Models Store More Features Than Neurons
Position bias: A judge’s tendency to prefer whichever answer it sees first (or second) regardless of quality. Countered by scoring both orders.Source: Large Language Models are not Fair Evaluators (Wang et al., 2023)Explained in Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift
Position interpolation: Extending a RoPE model's context window by scaling every position down by the stretch factor, so long inputs reuse the position range the model was trained on, followed by a short fine-tune.Source: Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., 2023)Explained in Positional Encoding: How Transformers Know Word Order
Positional encoding: Information added to a Transformer so it can tell token order. Attention on its own ignores order, so models add position vectors to the input or change attention scores based on position.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Positional Encoding: How Transformers Know Word Order
Post-norm: The original 2017 Transformer layout, which normalizes after adding each sub-layer's output: LayerNorm(x + Sublayer(x)). It usually needs a learning-rate warm-up to train.Source: On Layer Normalization in the Transformer Architecture (Xiong et al., 2020)Explained in Multi-Head Attention and the Transformer Block
Post-training quantization: Quantizing a finished model using a small calibration set, with no retraining.Source: GPTQ (Frantar et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
Power cap: When a GPU's power draw reaches its limit, the driver lowers clocks to stay under it. nvidia-smi reports this as the SW Power Cap event reason.Source: nvidia-smi documentation (NVIDIA)Explained in Profiling and the Roofline: Finding What Limits a GPU Kernel
Power law: A relationship y = a·x^b where one quantity changes as a fixed power of another. It plots as a straight line on log-log axes, with the exponent as the slope.Source: Scaling Laws for Neural Language Models (Kaplan et al., 2020)Explained in Counting: Parameters, FLOPs, Memory and Scaling Laws
PPOProximal policy optimization: A reinforcement learning algorithm that improves a policy in small, clipped steps. The standard optimizer in classic RLHF.Source: Proximal Policy Optimization Algorithms (Schulman et al., 2017)Explained in From Base Model to Assistant: SFT, RLHF and DPO
PQProduct quantization: Compressing a vector by splitting it into pieces and replacing each piece with the ID of its nearest centroid, so it is stored as a short code instead of full floats.Source: Product Quantization for Nearest Neighbor Search (Jégou et al., 2011)
PR mirroring: Making a benchmark task by undoing a real merged fix in the current code, so the bug returns and the fix's tests become hidden tests.Source: SWE-smith (Yang et al., 2025)Explained in Coding Agents: How They Fix Real Bugs, Measured
Pre-norm: Placing layer normalization at the input of each attention and MLP sub-block, as GPT-2 did, so the residual path stays unnormalized. It trains more stably than the 2017 post-norm layout.Source: On Layer Normalization in the Transformer Architecture (Xiong et al., 2020)Explained in Multi-Head Attention and the Transformer Block
Pre-tokenization: The step before BPE merging that splits text into pieces, usually with a regular expression, so merges never cross word, number or punctuation boundaries.Source: Language Models are Unsupervised Multitask Learners (Radford et al., 2019)Explained in Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens
Predication: Running both sides of a short branch, with each thread's copy of an instruction switched on or off by a one-bit flag, instead of jumping.Source: CUDA Programming Guide: programming model (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Preemption: When the KV cache runs out of memory, the server pauses a running request and frees its cache, then later restores it by recomputing it with prefill or swapping it back from CPU memory.Source: vLLM: Optimization and TuningExplained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
Prefill: The first phase of generation: the model processes the whole prompt in parallel and builds the KV cache. It is compute-heavy and sets the time to first token.Source: Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (Agrawal et al., 2024)Explained in How LLM Inference Works: Prefill, Decode and the Memory Wall
Prefix caching: Reusing the KV cache of a prompt prefix that was already processed, such as a shared system prompt, so only the new part needs prefill.Source: Automatic Prefix Caching (vLLM docs)Explained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
Previous-token head: An attention head that mostly attends from each token to the one just before it, copying information forward one position. Induction heads rely on it.Source: In-context Learning and Induction Heads (Olsson et al., 2022)Explained in Attention Heads and Induction Heads, Measured in GPT-2
Privileged basis: A representation where an element-wise operation, such as an activation function, makes the individual coordinates (neurons) special, so features may align with them.Source: Toy Models of Superposition (Elhage et al., 2022)Explained in Superposition: How Models Store More Features Than Neurons
Probability distribution: A list of possible outcomes with a probability for each; none is negative and they add up to 1. An LLM outputs one over its vocabulary at every step.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Probability and the Chain Rule of Language
Process reward model: A reward model that scores each step of a solution, not only the final answer.Source: Let’s Verify Step by Step (Lightman et al., 2023)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
Projection: How far one vector reaches along another's direction: its dot product with the other's unit vector, equal to its length times the cosine of the angle.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Vectors and Dot Products: Length, Angle and Meaning
Prompt caching: A provider feature that stores a processed prompt prefix, so later requests that repeat it are cheaper and start faster.Source: Prompt caching (Claude API docs)Explained in Cutting LLM Cost by 70% Without Losing Quality
Prompt injection: Text inside content the model reads, such as a web page, email or tool result, that tries to override its instructions, for example to leak data or misuse tools.Source: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (Greshake et al., 2023)Explained in Defending Tool-Using Agents Against Prompt Injection by Design
Prompt lookup decoding: Speculative decoding without a draft model: it copies candidate tokens from matching text in the prompt, which works well when the output repeats the input.Source: Prompt Lookup Decoding (Apoorv Saxena)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
PTX: Parallel Thread Execution, NVIDIA's virtual GPU instruction set. Compilers emit it, and the ptxas assembler turns it into SASS machine code.Source: PTX ISA (NVIDIA)Explained in Triton: Writing GPU Kernels in Python, Measured
PyTorch dispatcher: The PyTorch layer that picks which implementation of an operation runs (CPU, CUDA, autograd and so on) based on the inputs.Source: Let's talk about the PyTorch dispatcher (Edward Yang, 2020)Explained in From torch.matmul to Silicon: What Happens When PyTorch Runs

Q

QK circuit: The product of one attention head's query and key matrices: a model-width matrix, of rank at most the head size, that scores any pair of token vectors directly.Source: A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)Explained in The Math of Attention: Scaling, Masks, Jacobians and Low Rank
QK-norm: Normalizing each attention head's query and key vectors before their dot product, so attention scores can't grow large enough to make training diverge.Source: Scaling Vision Transformers to 22 Billion Parameters (Dehghani et al., 2023)Explained in The Modern LLM Block: What Changed Since 2017
QLoRA: LoRA trained on top of a base model stored in 4 bits (NF4), so large models can be fine-tuned on a single GPU.Source: QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)Explained in Fine-Tuning a Small Model with LoRA on 5,000 Examples
QPSQueries per second: Request rate, usually quoted at peak, which sizes the servers and indexes you need.Source: Queries per second (Wikipedia)
Quantization: Storing model weights (and sometimes activations or the KV cache) in fewer bits, such as 8 or 4 instead of 16, to cut memory and speed up inference at a small quality cost.Source: A Survey of Quantization Methods for Efficient Neural Network Inference (Gholami et al., 2021)Explained in Quantization: Running LLMs in 8 and 4 Bits

R

Radix select: Finding the k-th largest value by bucketing on a few bits at a time, as PyTorch's topk does, without sorting.Source: PyTorch TensorTopK.cu (v2.11.0)Explained in Reductions and Normalization Kernels: Summing Across Threads
Radix tree: A compact prefix tree whose edges hold runs of tokens. SGLang stores cached token sequences in one so requests can find their longest cached prefix.Source: SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)Explained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
RadixAttention: SGLang’s prefix cache: KV caches are kept in a radix tree, so any request that shares a prefix with an earlier one reuses its cache automatically.Source: SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)Explained in Prefix Caching and KV Cache Offloading: Never Prefill Twice
RAGRetrieval-augmented generation: Answering with an LLM after first retrieving relevant passages from your own documents and adding them to the prompt, so the answer is based on that content.Source: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)Explained in Designing a RAG System for 10 Million Documents
Rank (of a matrix): The number of independent directions a matrix's outputs can point in. It is at most the smaller of its two sizes, and a product of matrices can't exceed the lowest rank among them.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Matrices as Transformations, and Why Neural Networks Need a Bend
Rate matching: Sizing the prefill and decode pools of a disaggregated deployment so that prompts are prefilled about as fast as the decode pool can take them on, and adjusting the ratio as traffic changes.Source: Beyond the Buzz: A Pragmatic Take on Inference Disaggregation (Mitra et al., 2025)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
RDMARemote direct memory access: Moving data straight between the memory of two machines without involving their CPUs. Used to move KV caches and tensors quickly between GPU servers.Source: Remote direct memory access (Wikipedia)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
ReAct: An agent pattern that alternates reasoning steps with actions (tool calls) and their observations, looping until the task is done.Source: ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)Explained in Building Tool-Using Agents That Work Reliably
Reasoning model: An LLM trained, usually with reinforcement learning on checkable tasks, to write a long chain of thought before answering.Source: DeepSeek-R1 (DeepSeek-AI, 2025)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
Recall@k: The share of all relevant items that appear in the top k results. For RAG it asks: was the passage with the answer retrieved at all?Source: Introduction to Information Retrieval: evaluating ranked results (Manning et al.)Explained in Adding a Reranker and Proving It Helped
Reduce-scatter: A collective operation that sums a tensor across GPUs and leaves each GPU one slice of the total. It is the first half of a ring all-reduce, and FSDP uses it to reduce gradients.Source: NCCL User Guide: Collective Operations (NVIDIA)Explained in Many GPUs: NVLink, NCCL and Collectives
Redundant experts: Extra copies of frequently chosen MoE experts placed on other GPUs, so popular experts don't make one GPU the bottleneck.Source: DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)Explained in Case Study: DeepSeek's Inference Stack and DeepInfer
Reflexion: An agent method where, after a failed attempt, the model writes a short lesson about what went wrong, and the next attempt starts with that lesson in its prompt.Source: Reflexion (Shinn et al., 2023)Explained in Reflection and Self-Correction in Agents, Measured
Refusal direction: A single direction in a chat model's residual stream, the mean activation on harmful requests minus that on harmless ones. Removing it stops most refusals; adding it makes the model refuse harmless requests.Source: Refusal in Language Models Is Mediated by a Single Direction (Arditi et al., 2024)Explained in Steering With Activations: The Refusal Direction, Measured
Register blocking: Giving each GPU thread a small tile of outputs, so every value it loads into registers feeds several multiply-adds.Source: Efficient GEMM in CUDA (NVIDIA CUTLASS docs)Explained in Writing a Fast Matmul in CUDA, Step by Step
Register spill: When a kernel needs more registers than it can get, the compiler stores values in per-thread local memory, which physically lives in DRAM and is cached in L1 and L2.Source: CUDA Programming Guide: writing CUDA kernels (NVIDIA)Explained in The GPU Memory Hierarchy: Registers, Caches and DRAM, Measured
Rejection sampling: Drawing proposals from an easy distribution and keeping each with a probability that makes the kept ones follow a target distribution. In LLM fine-tuning, the name also covers sampling several answers per prompt and keeping the one a reward model scores highest.Source: The Llama 3 Herd of Models (Llama Team, Meta, 2024)Explained in From Base Model to Assistant: SFT, RLHF and DPO
ReLURectified linear unit: An activation function that keeps positive inputs and sets negative ones to zero: max(0, x).Source: Deep Learning, Chapter 6 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
Repetition penalty: A sampling setting that lowers the logits of tokens already generated, to discourage loops.Source: Advanced usage (OpenAI API docs)Explained in Encoders, Decoders and GPT, and How Sampling Works
Replacement model: A version of a model with its MLPs swapped for transcoders. The local version also freezes attention on one prompt and adds error terms so it matches the real model exactly.Source: Circuit Tracing (Ameisen et al., 2025)Explained in Circuits and Attribution Graphs, Traced on a Laptop GPU
Repo map: A compact listing of a repository's files and the classes and functions they define, given to a coding model so it can find code without reading every file.Source: Repository map (Aider docs)Explained in Coding Agents: How They Fix Real Bugs, Measured
Reranker: A second-stage model that re-scores a short list of retrieved candidates more accurately (and more slowly) than the first search, then reorders them.Source: Rerank overview (Cohere docs)Explained in Adding a Reranker and Proving It Helped
Residual connection: A path that adds a layer's input to its output, so the layer only has to learn a correction. It lets very deep networks train.Source: Deep Residual Learning for Image Recognition (He et al., 2015)Explained in Multi-Head Attention and the Transformer Block
Residual stream: The running vector for each token in a Transformer: the embedding plus everything each attention layer and MLP has added to it. Layers read from it and write to it by addition.Source: A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)Explained in Multi-Head Attention and the Transformer Block
Residual vector quantization: Quantizing a vector with a chain of codebooks, where each codebook encodes the error left by the ones before it, so more codebooks give finer detail.Source: SoundStream (Zeghidour et al., 2021)Explained in Multimodal Models: How LLMs See and Hear
reStructuredText: A plain-text markup language, used by Sphinx, for writing documentation that stays readable as source.Source: reStructuredText (Docutils)Explained in An Agent End to End: A Docs-Maintenance Agent, Measured
Reverse-mode autodiff: Automatic differentiation that runs the chain rule from the output back to the inputs, giving the gradient of one output with respect to every input in a single pass. Forward mode goes the other way and needs one pass per input.Source: Automatic differentiation in machine learning: a survey (Baydin et al., 2015)Explained in Backpropagation by Hand: From the Chain Rule to Autograd
Reward hacking: When optimization finds outputs that score highly on an imperfect learned reward but that people don't actually prefer.Source: Scaling Laws for Reward Model Overoptimization (Gao et al., 2022)Explained in From Base Model to Assistant: SFT, RLHF and DPO
Reward model: A model that outputs a single score for a prompt and answer, trained on pairs where people picked the better answer. RLHF optimizes against it.Source: Training language models to follow instructions with human feedback (Ouyang et al., 2022)Explained in From Base Model to Assistant: SFT, RLHF and DPO
Ring attention: Splitting one long sequence across GPUs, each holding a chunk of queries, keys and values, and passing key-value blocks around a ring while computing.Source: Ring Attention with Blockwise Transformers (Liu et al., 2023)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
RLAIFReinforcement learning from AI feedback: Preference tuning where a model, guided by written principles, judges which answer is better in place of human labelers. Anthropic's Constitutional AI introduced it.Source: Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022)Explained in From Base Model to Assistant: SFT, RLHF and DPO
RLHFReinforcement learning from human feedback: Fine-tuning a model with reinforcement learning to maximize a reward model trained on human preference comparisons, usually with a penalty for drifting from the starting model.Source: Training language models to follow instructions with human feedback (Ouyang et al., 2022)Explained in From Base Model to Assistant: SFT, RLHF and DPO
RLVRReinforcement learning with verifiable rewards: Reinforcement learning where the reward comes from a program that checks correctness, such as a math answer, instead of a learned reward model.Source: Tülu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024)Explained in From Base Model to Assistant: SFT, RLHF and DPO
RMSNormRoot mean square layer normalization: A simpler layer normalization that divides each vector by the root mean square of its numbers, then multiplies by a learned gain. It skips subtracting the mean and has no bias.Source: Root Mean Square Layer Normalization (Zhang & Sennrich, 2019)Explained in The Modern LLM Block: What Changed Since 2017
RMSProp: An optimizer that divides each weight's gradient by the root mean square of its recent gradients, giving every weight its own step size.Source: Neural Networks for Machine Learning, lecture 6 (Hinton et al., 2012)Explained in Optimizers: From Gradient Descent to AdamW
RNNRecurrent neural network: A neural network that reads a sequence one step at a time, carrying a running summary forward. The main approach to language before the Transformer, hard to parallelize.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in How a Large Language Model Works: The Big Picture
ROCm: AMD's open software platform for GPU computing: compilers, HIP and libraries such as rocBLAS, hipBLASLt and RCCL, the counterparts of NVIDIA's CUDA libraries.Source: ROCm documentation (AMD)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Rolling buffer cache: A fixed-size KV cache for sliding-window layers where each new token overwrites the oldest slot, so memory per user stays constant.Source: Mistral 7B (Jiang et al., 2023)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
ROMERank-One Model Editing: A method that changes one fact a language model recalls with a rank-one update to one MLP layer's output weights.Source: Locating and Editing Factual Associations in GPT (Meng et al., 2022)Explained in Activation Patching and Causal Tracing, Measured in GPT-2
Roofline: A model of hardware performance: speed is capped either by memory bandwidth or by peak compute. The ridge point is the arithmetic intensity where the two caps meet (about 295 FLOPs per byte on an H100).Source: LLM Inference Unveiled: Survey and Roofline Model Insights (Yuan et al., 2024)Explained in Profiling and the Roofline: Finding What Limits a GPU Kernel
RoPE: Rotary position embedding: marks each token's position by rotating its query and key vectors by an angle that depends on that position.Source: RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)
Rotation matrix: A 2 × 2 matrix of cosines and sines that turns every vector by an angle θ without changing its length. Its transpose turns it back.Source: RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)Explained in Rotations, Complex Numbers and the Math of RoPE
Round to nearest, ties to even: IEEE 754's default rounding: a value exactly halfway between two floats goes to the one whose last bit is 0, so rounding has no upward bias.Source: What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg, 1991)Explained in Floating Point and Quantization: The Math of Fewer Bits
Router (MoE): In a mixture-of-experts layer, a small linear layer that scores every expert for each token and sends the token to the highest-scoring few.Source: Outrageously Large Neural Networks (Shazeer et al., 2017)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
Router z-loss: A training penalty on large router logits that keeps mixture-of-experts training numerically stable.Source: ST-MoE (Zoph et al., 2022)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
RRFReciprocal rank fusion: A way to merge ranked lists: each document scores the sum of 1/(k + rank) over the lists it appears in, with k usually 60. It uses only ranks, so scores on different scales need no calibration.Source: Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (Cormack et al., 2009)Explained in BM25 vs Dense vs Hybrid Search: When to Use Each
RULER: An NVIDIA benchmark of 13 synthetic long-context tasks that measures a model's effective context length.Source: RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., 2024)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction

S

Safetensors: A file format for model weights that starts with a JSON header listing every tensor's name, type and shape, followed by the raw numbers.Source: Safetensors (Hugging Face docs)Explained in Counting: Parameters, FLOPs, Memory and Scaling Laws
SASS: The machine instructions an NVIDIA GPU actually executes, produced from PTX by the ptxas assembler and shown by cuobjdump -sass.Source: CUDA Binary Utilities (NVIDIA)Explained in Triton: Writing GPU Kernels in Python, Measured
Scaled dot-product attention: The Transformer's attention formula, softmax(QKᵀ/√d_k)V: score every query against every key, divide by the square root of the key size, turn each row into weights with softmax, and average the values.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Attention Explained: Queries, Keys and Values
Scaling laws: Measured power-law relationships between a model's loss and its parameter count, training tokens and compute. Labs use them to predict a large model's results from small runs.Source: Scaling Laws for Neural Language Models (Kaplan et al., 2020)Explained in Pretraining: Data, Scaling Laws and Training at Scale
seccomp: A Linux kernel filter that limits which system calls a process may make. Docker's default profile blocks around 44 of them.Source: Seccomp security profiles for DockerExplained in Sandboxing and Permissions for AI Agents, Measured
Selective batching: Batching the token-independent layers across all requests while running attention separately per request, so requests at different stages can share one step.Source: Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., 2022)Explained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
Self-attention: Attention in which every token in a sequence compares itself with the other tokens of the same sequence and mixes in their information. The core operation of a Transformer.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Attention Explained: Queries, Keys and Values
Self-conditioning: A model becoming more likely to make mistakes once its context contains its own earlier mistakes.Source: The Illusion of Diminishing Returns (Sinha et al., 2025)Explained in Long-Horizon Reliability: Compounding Errors, Retries and Checkpoints
Self-consistency: Sampling several chains of thought for one question and taking a majority vote on their final answers.Source: Self-Consistency Improves Chain of Thought Reasoning (Wang et al., 2022)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
Self-Refine: A prompting method where one model writes an output, critiques it and rewrites it using its own feedback, for a few rounds, with no training or tools.Source: Self-Refine (Madaan et al., 2023)Explained in Reflection and Self-Correction in Agents, Measured
Semantic cache: A cache that returns a stored answer when a new query means the same as an earlier one, judged by embedding similarity instead of exact text.Source: GPTCache (Zilliz)Explained in Designing a Semantic Cache with Hit-Rate Tracking
SentencePiece: Google's open-source tokenizer library that trains directly on raw text and treats spaces as ordinary symbols, so it works the same for languages with or without spaces between words.Source: SentencePiece (Kudo and Richardson, 2018)Explained in Tokenization: How LLMs Turn Text into Tokens
seq2seqSequence to sequence: A model that maps one sequence to another of possibly different length, such as a sentence to its translation, usually with an encoder and a decoder.Source: Sequence to Sequence Learning with Neural Networks (Sutskever et al., 2014)Explained in Reading "Attention Is All You Need", the 2017 Transformer Paper
Sequence-level distillation: Training a student on text a teacher generated rather than on the teacher's probabilities, so the two need not share a vocabulary.Source: Sequence-Level Knowledge Distillation (Kim and Rush, 2016)Explained in Build an LLM From Scratch: Distillation and Synthetic Data
Server tool: A tool that the Claude API runs itself inside the request, such as web search or web fetch, returning the call and its result in the same response, so your code does not execute it.Source: Server tools (Claude API docs)Explained in Research Agents: Web Search, Fetch and Citations, Measured
SFTSupervised fine-tuning: Training a model further on example inputs paired with the outputs you want.Source: Training language models to follow instructions with human feedback (Ouyang et al., 2022)Explained in From Base Model to Assistant: SFT, RLHF and DPO
SGDStochastic gradient descent: Gradient descent where each step uses the average slope over a random sample (a mini-batch) of training examples instead of the full dataset.Source: Deep Learning, Chapter 8 (Goodfellow, Bengio & Courville, 2016)Explained in Neural Networks: Weights, Gradient Descent and Backpropagation
SGLang: An open-source LLM serving engine with strong prefix caching (RadixAttention) and fast structured output.Source: SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)Explained in Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization
Shadow traffic: Running a new version on copies of real requests without showing its output to users, so it can be compared safely.Source: Shadow tests (Amazon SageMaker docs)Explained in Rolling Back a Bad Prompt Change in Production
Shared expert: An expert in a mixture-of-experts layer that every token uses, alongside the routed experts, to hold common knowledge.Source: DeepSeekMoE (Dai et al., 2024)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
Shingle: A run of n consecutive words (or characters) from a document. A document becomes the set of its shingles, so two documents can be compared as sets.Source: Mining of Massive Datasets, Chapter 3 (Leskovec, Rajaraman and Ullman)Explained in The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop
Sigmoid: The function 1/(1 + e^-z), which squeezes any number into the range 0 to 1. Its slope is sigmoid(z) times (1 - sigmoid(z)).Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Derivatives, Gradients and the Chain Rule
SiLUSigmoid linear unit: The activation z × sigmoid(z): close to ReLU, but smooth. It is the gate activation in SwiGLU.Source: Searching for Activation Functions (Ramachandran et al., 2017)Explained in The Modern LLM Block: What Changed Since 2017
SIMT: Single instruction, multiple threads: each thread runs its own code path, but a warp executes together, masking off threads that aren't on the current path.Source: CUDA Programming Guide: programming model (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Singular value: How much a matrix stretches vectors along one of its special perpendicular directions. Sorted largest first, their squares add up to the sum of the squares of all the matrix's entries.Source: The Approximation of One Matrix by Another of Lower Rank (Eckart & Young, 1936)Explained in Low Rank and the SVD: How Much of a Matrix Really Matters
Sleeper agent: A model deliberately trained with a hidden trigger that switches it to bad behavior, used to test whether safety training and detection methods catch it.Source: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Hubinger et al., 2024)Explained in Interpretability for Safety: Lie Detectors, Sycophancy and Audits
Sliding window attention: Attention in which each token sees only the last W tokens instead of the whole context, so the KV cache stops growing past W.Source: Mistral 7B (Jiang et al., 2023)Explained in The Modern LLM Block: What Changed Since 2017
SLOService-level objective: A target for how a service should behave, such as “p95 latency under 2 seconds”, that you measure and alert on.Source: Service Level Objectives (Google SRE Book)
SMStreaming multiprocessor: The basic building block of an NVIDIA GPU: a unit with its own cores, tensor cores, registers, shared memory and scheduler. An H100 SXM has 132.Source: NVIDIA Hopper Architecture In-Depth (NVIDIA, 2022)Explained in How GPUs Work for LLMs: Cores, Memory and Number Formats
SMESubject-matter expert: A person with deep domain knowledge, such as a lawyer or support lead, who writes or checks the labels in an eval set.Source: Subject-matter expert (Wikipedia)
SmoothQuant: A post-training method that moves activation outliers into the weights by dividing each activation channel by a factor and multiplying the matching weights by it, so both can run in 8 bits (W8A8).Source: SmoothQuant (Xiao et al., 2022)Explained in Quantization: Running LLMs in 8 and 4 Bits
SnapKV: A method that compresses a prompt's KV cache after prefill, keeping per head the positions that the prompt's final tokens attend to most.Source: SnapKV (Li et al., 2024)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Soft targets: A teacher model's full probability distribution over outputs, used as a student's training target instead of the single correct label.Source: Distilling the Knowledge in a Neural Network (Hinton et al., 2015)Explained in Build an LLM From Scratch: Distillation and Synthetic Data
Softmax: A function that turns a list of scores into probabilities that add up to 1, by exponentiating each score and dividing by the total. Bigger scores get much bigger shares.Source: Transformers, the tech behind LLMs (3Blue1Brown, 2024)Explained in How a Large Language Model Works: The Big Picture
Sparse attention: Attention where each query reads a selected subset of past tokens, chosen by fixed patterns or learned scores, instead of all of them.Source: Native Sparse Attention (Yuan et al., 2025)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Sparse autoencoderSAE: A two-layer network trained to rebuild a model's activations from a very wide set of features, only a few active at once, so that each feature is easier to interpret.Source: Towards Monosemanticity (Bricken et al., 2023)Explained in Sparse Autoencoders: Pulling Features Out of a Model
Special token: A vocabulary entry that never comes from ordinary text and acts as a control signal, such as an end-of-text marker or a marker for where a chat message starts or ends.Source: Meta Llama 3 prompt formatExplained in Tokenization: How LLMs Turn Text into Tokens
Spectrogram: A picture of sound over time: for each short window it records how much energy falls in each frequency band. Mel bands are spaced the way human hearing is.Source: Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022)Explained in Multimodal Models: How LLMs See and Hear
Speculative decoding: A small draft model guesses several tokens ahead and the large model checks them all in one pass. Generation gets faster while the output stays the same as the large model’s.Source: Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Speculative sampling: The acceptance rule that keeps speculative decoding exact when sampling: keep a drafted token with probability min(1, p/q), otherwise resample from the leftover probability max(0, p − q).Source: Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Sphinx: The standard documentation generator for Python projects. It builds websites and other formats from text sources and resolves cross-references between pages and code objects.Source: Sphinx documentationExplained in An Agent End to End: A Docs-Maintenance Agent, Measured
SPLADE: A learned sparse retriever: a neural model expands text into weighted keywords, so search runs on an inverted index like BM25 but also matches related terms.Source: SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (Formal et al., 2021)Explained in BM25 vs Dense vs Hybrid Search: When to Use Each
Split-K: Splitting a matrix multiply's inner dimension across thread blocks and summing the partial results, to fill the GPU when the output is small.Source: Efficient GEMM in CUDA (NVIDIA CUTLASS docs)Explained in Writing a Fast Matmul in CUDA, Step by Step
Splitwise: A 2023 Microsoft research system that runs prompt processing and token generation on separate machines, possibly of different GPU types, to raise throughput per dollar or per watt.Source: Splitwise (Patel et al., 2023)Explained in Disaggregated Serving and llm-d: Splitting Prefill from Decode
SRAM: Fast memory built into the GPU chip (registers, shared memory and caches). It is about an order of magnitude faster than HBM but holds megabytes, not gigabytes.Source: FlashAttention (Dao et al., 2022)Explained in How GPUs Work for LLMs: Cores, Memory and Number Formats
SSEServer-sent events: A simple web standard for a server to stream text events to a browser over one long-lived HTTP response. The usual way to stream LLM output.Source: HTML Standard: Server-sent events (WHATWG)Explained in Streaming LLM Responses over SSE and Surviving Disconnects
State-space model: A sequence layer that folds each token into a fixed-size state, like a recurrent network, so generation needs no growing KV cache.Source: Mamba (Gu and Dao, 2023)Explained in Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
Static batching: Running a fixed group of requests together until the longest one finishes, so the slots of finished requests sit idle and new requests wait.Source: Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., 2022)Explained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
Static embedding: A single fixed vector per token that is the same in every context, like a row of an LLM's input table or a word2vec vector. The Transformer blocks turn it into a contextual vector.Source: Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013)Explained in Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
Static KV cache: A KV cache preallocated at a fixed maximum length, so every decode step has the same tensor shapes and memory addresses, which CUDA graphs and torch.compile need.Source: Accelerating Generative AI with PyTorch II: GPT, Fast (PyTorch)Explained in One Token, Kernel by Kernel: A Decode Step on a Laptop GPU
Statistical power: The chance that an experiment detects a difference that is really there. 80% is the usual target, and it rises with the number of tasks and the size of the gap.Source: Adding Error Bars to Evals (Miller, 2024)Explained in Evaluating Agents: pass@k, pass^k and Run-to-Run Noise, Measured
Steering vector: A direction added to a model's activations during generation to push its output toward a concept or style, such as "Love" minus "Hate".Source: Steering Language Models With Activation Engineering (Turner et al., 2023)Explained in Steering With Activations: The Refusal Direction, Measured
Streamable HTTP: The MCP transport for remote servers: one HTTP endpoint that takes a POST per message and answers with JSON or an SSE stream.Source: MCP specification: Streamable HTTPExplained in MCP: Connecting Agents to Tools
Strict tool use: An API setting that compiles a tool's JSON Schema into a grammar, so the model can only produce tool inputs that match it. It guarantees shape, not correct values.Source: Strict tool use (Claude API docs)Explained in How Tool Calling Works Inside the Model
Structured output: A model provider feature that makes the response follow a JSON schema you supply.Source: Structured Outputs guide (OpenAI)Explained in Guaranteeing Valid JSON Output from LLMs
Structured sparsity: A pattern where at least two of every four consecutive values are zero, which tensor cores can skip for twice the throughput. Datasheet figures marked "with sparsity" assume it.Source: Structured Sparsity in the NVIDIA Ampere Architecture (NVIDIA)Explained in How GPUs Work for LLMs: Cores, Memory and Number Formats
Sub-agent: An agent started by another agent to handle one subtask in its own context window, returning only a report to the agent that started it.Source: Subagents (Claude Code docs)Explained in Multi-Agent Systems: Orchestrators, Sub-agents and A2A, Measured
Subnormal number: A float smaller than the smallest normal value, stored without the hidden leading 1, which fills the gap down to zero at reduced precision.Source: What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg, 1991)Explained in Floating Point and Quantization: The Math of Fewer Bits
Superposition: A model representing more features than it has dimensions by giving each one a nearly perpendicular direction. It works when features are rarely active at the same time.Source: Toy Models of Superposition (Elhage et al., 2022)Explained in Superposition: How Models Store More Features Than Neurons
SVD: The singular value decomposition: any matrix written as a rotation, a stretch along perpendicular axes, and another rotation (U Σ Vᵀ). Keeping the largest stretches gives the best low-rank approximation.Source: The Approximation of One Matrix by Another of Lower Rank (Eckart & Young, 1936)Explained in Low Rank and the SVD: How Much of a Matrix Really Matters
SWE-bench: A benchmark of 2,294 real GitHub issues from 12 Python repositories. A patch counts only if hidden tests from the real fix pass and the existing tests still pass.Source: SWE-bench (Jimenez et al., 2023)Explained in Coding Agents: How They Fix Real Bugs, Measured
SWE-Gym: The first training environment for coding agents: 2,438 executable tasks from real GitHub issues in 11 Python repositories, with Docker images and tests.Source: SWE-Gym (Pan et al., 2024)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
SWE-smith: A pipeline that makes coding-agent training tasks by creating bugs inside 128 repositories, giving 50,000 tasks that share one environment per repository.Source: SWE-smith (Yang et al., 2025)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
SwiGLU: A feed-forward layer design that multiplies two parallel projections, one passed through the Swish activation, before projecting back. Llama models use it instead of ReLU or GELU.Source: GLU Variants Improve Transformer (Shazeer, 2020)Explained in The Modern LLM Block: What Changed Since 2017
Switch Transformer: Google's 2021 mixture-of-experts Transformer that sends each token to a single expert. It scaled to 1.6 trillion parameters.Source: Switch Transformers (Fedus, Zoph & Shazeer, 2021)Explained in Mixture of Experts: How Giant Models Use a Fraction of Their Weights
Swizzling: Permuting where data chunks sit in shared memory, often by XOR-ing address bits, so threads reading a tile's rows or columns hit different banks without padding.Source: CuTe swizzle.hpp (NVIDIA CUTLASS)Explained in Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
Sycophancy: A model's tendency to tell users what they appear to want to hear rather than what is true, partly learned from human preference data.Source: Towards Understanding Sycophancy in Language Models (Sharma et al., 2023)Explained in From Base Model to Assistant: SFT, RLHF and DPO
Symmetric token replacement: A corruption for activation patching that swaps key tokens for similar real ones of the same length, so the corrupted prompt stays like ordinary text.Source: Towards Best Practices of Activation Patching (Zhang & Nanda, 2023)Explained in Activation Patching and Causal Tracing, Measured in GPT-2
Systolic array: A grid of multiply-accumulate cells that pass operands and partial sums to their neighbors every clock cycle, so data is reused without going back to memory. TPUs and Trainium build their matrix units this way.Source: In-Datacenter Performance Analysis of a Tensor Processing Unit (Jouppi et al., 2017)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators

T

T5Text-to-Text Transfer Transformer: Google's 2019 encoder-decoder Transformer that casts every task as text in, text out and is pretrained by span corruption.Source: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019)Explained in Encoders, Decoders and GPT, and How Sampling Works
Temperature: A sampling setting that divides the logits before softmax. Below 1 makes the top token more likely; above 1 spreads probability toward less likely tokens.Source: How to generate text (von Platen, Hugging Face, 2020)Explained in How a Large Language Model Works: The Big Picture
Tensor core: A GPU unit that multiplies two small matrices and adds the result to a third in one operation. Nearly all of an LLM's arithmetic runs on tensor cores.Source: NVIDIA Hopper Architecture In-Depth (NVIDIA, 2022)Explained in How GPUs Work for LLMs: Cores, Memory and Number Formats
Tensor memory: Blackwell's on-chip memory dedicated to tensor cores. A tcgen05.mma instruction, issued by one thread, accumulates its results there instead of in registers.Source: PTX ISA (NVIDIA)Explained in Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
Tensor parallelism: Splitting each layer’s weight matrices across several GPUs that compute in lockstep, so a model too big for one GPU runs as a single replica. Needs fast links such as NVLink.Source: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., 2019)Explained in Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs
Test-time compute: Computation spent while answering, such as longer reasoning or sampling many answers, as opposed to compute spent in training.Source: Scaling LLM Test-Time Compute Optimally (Snell et al., 2024)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
TF32: A tensor-core math mode that rounds FP32 inputs to 19 bits (8 exponent, 10 mantissa) and adds the results up in FP32. It is not a storage format.Source: Accelerating AI Training with TF32 Tensor Cores (NVIDIA)Explained in How GPUs Work for LLMs: Cores, Memory and Number Formats
TFLOPS / PFLOPS: Trillions (TFLOPS) or quadrillions (PFLOPS) of floating-point operations per second: a GPU’s peak compute speed.Source: NVIDIA H100 Tensor Core GPU specifications
Thinking tokens: Tokens a model generates while reasoning before its visible answer. APIs bill them as output tokens even when they are hidden.Source: Reasoning models (OpenAI API docs)Explained in Reasoning Models: Chain of Thought, RL and Test-Time Compute
Thread block: A group of GPU threads scheduled together on one streaming multiprocessor, sharing its on-chip shared memory.Source: FlashAttention-3 (Shah et al., 2024)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
tiktoken: OpenAI's open-source library for the byte-level BPE tokenizers its models use, such as cl100k_base (GPT-4) and o200k_base (GPT-4o).Source: openai/tiktoken (GitHub)Explained in Tokenization: How LLMs Turn Text into Tokens
Tiling: Splitting a large matrix computation into small blocks (tiles) that fit in on-chip memory, so each block is loaded once and reused.Source: FlashAttention (Dao et al., 2022)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
tinygemm: PyTorch's built-in INT4 weight-only matrix-multiply kernel (_weight_int4pack_mm). It is fast at batch sizes of a few rows and slow for large batches.Source: int4mm.cu (PyTorch source)Explained in Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
TinyStories: A dataset of about 2 million short, simple stories written by GPT-3.5 and GPT-4, used to train very small language models that still write coherent English.Source: TinyStories (Eldan and Li, 2023)Explained in Build an LLM From Scratch: What a Laptop Can Train
TMATensor Memory Accelerator: Tensor Memory Accelerator: a Hopper GPU unit that copies tiles between HBM and shared memory on its own, so loading and computing can overlap.Source: FlashAttention-3 (Shah et al., 2024)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
TNRTrue negative rate: The share of real negatives that a check correctly lets through. Also called specificity.Source: Sensitivity and specificity (Wikipedia)
Token budget: The most tokens, prompt and generated combined, that a serving engine processes in one forward pass. It trades time to first token against time per output token.Source: Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (Agrawal et al., 2024)Explained in Continuous Batching and Chunked Prefill: Keeping the GPU Full
Tokenization premium: How many tokens a language needs to express the same content as English with a given tokenizer. A premium of 3 means three times the tokens.Source: Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al., 2023)Explained in Tokenization: How LLMs Turn Text into Tokens
Tokenizer: The fixed program that splits text into tokens (whole words or word pieces) from a vocabulary and maps each one to an ID number the model can read.Source: Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2016)Explained in Tokenization: How LLMs Turn Text into Tokens
Tool call: A structured request from the model to run a function, such as a search, database query or API call, that your code executes and returns the result of.Source: Tool use (Claude API docs)Explained in Building Tool-Using Agents That Work Reliably
Tool poisoning: An attack where an MCP server hides instructions in its tool descriptions, which the model reads but users usually do not.Source: MCP Security Notification: Tool Poisoning Attacks (Invariant Labs, 2025)Explained in MCP: Connecting Agents to Tools
Tool search: A Claude API feature where tool definitions marked defer_loading stay out of the prompt until the model finds them with a search tool.Source: Tool search tool (Claude API docs)Explained in MCP: Connecting Agents to Tools
Toolformer: A 2023 Meta model that taught itself to call tools by inserting candidate API calls into text and keeping only those that lowered its loss on the following tokens.Source: Toolformer (Schick et al., 2023)Explained in Training Models to Be Agents: Trajectories, Environments, Rewards
Top-k sampling: Sampling only from the k most likely next tokens, after renormalizing their probabilities.Source: Hierarchical Neural Story Generation (Fan et al., 2018)Explained in Encoders, Decoders and GPT, and How Sampling Works
TopK SAE: A sparse autoencoder that keeps only the k largest latent activations per token, fixing sparsity without an L1 penalty.Source: Scaling and evaluating sparse autoencoders (Gao et al., 2024)Explained in Sparse Autoencoders: Pulling Features Out of a Model
torch.compile: PyTorch 2's compiler entry point: TorchDynamo captures Python code as a graph and TorchInductor fuses operations and generates kernels (Triton on GPUs).Source: torch.compiler (PyTorch docs)Explained in FlashAttention and Fused Kernels: Attention Without the n×n Matrix
TorchDynamo: The part of torch.compile that rewrites Python bytecode at run time to capture PyTorch operations as a graph, with guards that decide when to reuse or recompile it.Source: PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation (Ansel et al., 2024)Explained in Kernel Fusion, torch.compile and CUDA Graphs, Measured
TorchInductor: PyTorch's default compiler backend for torch.compile. It fuses operations and generates Triton kernels for GPUs, calling libraries like cuBLAS for matrix multiplies.Source: PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation (Ansel et al., 2024)Explained in Kernel Fusion, torch.compile and CUDA Graphs, Measured
TPOTTime per output token: The average gap between generated tokens after the first one. It sets how fast the answer streams.Source: LLM inference benchmarking metrics (NVIDIA NIM docs)Explained in Reducing Time to First Token in a Chat App
TPRTrue positive rate: The share of real positives that a check correctly catches. Also called recall or sensitivity.Source: Sensitivity and specificity (Wikipedia)
TPU: Tensor processing unit: Google's AI accelerator, built around systolic matrix units, linked chip to chip in a torus, and programmed through the XLA compiler.Source: Cloud TPU system architecture (Google)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Trainium: AWS's AI accelerator chips, each with eight NeuronCores built around 128 × 128 systolic Tensor Engines and linked by NeuronLink.Source: Trainium3 architecture (AWS Neuron docs)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Transcoder: A sparse network trained to stand in for a model's MLP layer: it reads the MLP's input and predicts its output through a wide layer of features, few of them active at once.Source: Transcoders Find Interpretable LLM Feature Circuits (Dunefsky et al., 2024)Explained in Circuits and Attribution Graphs, Traced on a Laptop GPU
Transformer: The neural network architecture behind modern LLMs: a stack of identical blocks, each combining attention (tokens sharing information) with a small feed-forward network. Introduced in 2017.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in How a Large Language Model Works: The Big Picture
TransformerLens: An open-source library that loads language models with a named hook at every internal activation, so activations can be cached, read or replaced while the model runs.Source: TransformerLens (GitHub)Explained in Opening the Black Box: What Interpretability Can Explain
Tree attention: An attention mask that lets one forward pass verify a tree of candidate continuations at once, each branch seeing only its own ancestors.Source: Medusa (Cai et al., 2024)Explained in Speculative Decoding: Guess Several Tokens, Check Them in One Pass
Triton: A language and compiler for writing GPU kernels in terms of tiles, from Python. torch.compile and many attention kernels use it. Not to be confused with NVIDIA's Triton Inference Server.Source: Triton (Tillet, Kung & Cox, MAPL 2019)Explained in Triton: Writing GPU Kernels in Python, Measured
TTFTTime to first token: How long a user waits from sending a request until the first token of the answer arrives. The latency people notice most in chat.Source: LLM inference benchmarking metrics (NVIDIA NIM docs)Explained in Reducing Time to First Token in a Chat App
TTLTime to live: How long a cached entry or record stays valid before it expires.Source: Time to live (Wikipedia)
Tuned lens: A logit lens with one small trained affine translator per layer, fitted so that each intermediate layer's decoded prediction matches the model's final output.Source: Eliciting Latent Predictions from Transformers with the Tuned Lens (Belrose et al., 2023)Explained in The Residual Stream and the Logit Lens

U

Underflow: When a number is too small for its floating-point format and rounds to 0. float32 cannot store positive values below about 1.4e-45.Source: numpy.finfo (NumPy docs)Explained in Probability and the Chain Rule of Language
Unembedding: The final layer of a language model: it takes the dot product of the last vector with one row per vocabulary token to produce a logit for every token. Also called the LM head.Source: Attention Is All You Need (Vaswani et al., 2017)Explained in Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
Unified memory: One pool of memory shared by the CPU and GPU, as on Apple silicon, so the GPU can use most of the system's RAM without copying data between separate pools.Source: Apple unleashes M1 (Apple, 2020)Explained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
Unigram tokenizer: A tokenizer training method that starts with a large set of candidate pieces and repeatedly removes the ones that matter least, keeping a probability for each piece.Source: Subword Regularization (Kudo, 2018)Explained in Tokenization: How LLMs Turn Text into Tokens
Unit roundoff: The largest relative error from rounding a real number to the nearest float, 2^-(mantissa bits + 1). Machine epsilon, the gap after 1.0, is twice it.Source: What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg, 1991)Explained in Floating Point and Quantization: The Math of Fewer Bits
Unit vector: A vector of length exactly 1. Dividing any vector by its own length gives a unit vector pointing the same way, a step called normalization.Source: Mathematics for Machine Learning (Deisenroth, Faisal & Ong)Explained in Vectors and Dot Products: Length, Angle and Meaning
UTF-8: The standard way to store Unicode text as bytes: 1 byte for basic Latin letters and 2 to 4 bytes for other characters.Source: RFC 3629: UTF-8, a transformation format of ISO 10646Explained in Tokenization: How LLMs Turn Text into Tokens

V

Validation loss: A model's average next-token loss on text kept out of training, used to check that improvements in training loss carry over to unseen text.Source: Training Compute-Optimal Large Language Models (Hoffmann et al., 2022)Explained in Build an LLM From Scratch: The Pretraining Run
Vanishing gradient: Gradients that shrink toward zero (or grow without bound) as they pass back through many layers, because each layer multiplies them by a factor below (or above) 1.Source: Understanding the difficulty of training deep feedforward neural networks (Glorot & Bengio, 2010)Explained in Keeping Signals Stable: Initialization, Normalization and Residuals
Vision TransformerViT: A Transformer that reads an image as a sequence of small square patches, each flattened and multiplied by a learned matrix to make a vector.Source: An Image is Worth 16x16 Words (Dosovitskiy et al., 2020)Explained in Multimodal Models: How LLMs See and Hear
Visual token: A 28 by 28-pixel patch of an image, the unit Claude bills images in, so an image costs about (width / 28) times (height / 28) tokens, each rounded up.Source: Vision (Claude API docs)Explained in Computer-Use and Browser Agents: Pixels Versus the Page
vLLM: A widely used open-source LLM serving engine, known for PagedAttention and continuous batching.Source: vLLM documentationExplained in Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization
VRAMVideo RAM: A graphics card's own memory. A model runs at the GPU's full speed only if it fits there.Source: NVIDIA GeForce graphics cards comparedExplained in Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama

W

W8A8 / W4A16: Shorthand for quantization schemes: the number after W is the bits per weight and the number after A the bits per activation. W4A16 is weight-only 4-bit; W8A8 quantizes both to 8 bits.Source: "Give Me BF16 or Give Me Death"? (Kurtić et al., 2024)Explained in Quantization: Running LLMs in 8 and 4 Bits
Wafer-scale engine: Cerebras's processor made from a whole silicon wafer; the WSE-3 has 900,000 cores and 44 GB of on-chip SRAM.Source: Cerebras WSE-3 announcementExplained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Warmup-stable-decay schedule: A learning-rate schedule that warms up, holds the peak rate, and decays only at the end, so the training length need not be fixed in advance.Source: Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (Hägele et al., 2024)Explained in Build an LLM From Scratch: From Our Model to Frontier Models
Warp: A group of 32 GPU threads that the hardware issues one instruction to at a time.Source: CUDA Programming Guide: programming model (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Warp divergence: When threads of one warp take different branches, the warp runs each path in turn with the other threads idle.Source: CUDA Programming Guide: programming model (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Warp shuffle: A GPU instruction, such as __shfl_down_sync, that lets threads in one warp read each other's register values directly, without shared memory.Source: CUDA Programming Guide: C++ language extensions (NVIDIA)Explained in Reductions and Normalization Kernels: Summing Across Threads
Warp stall reason: Nsight Compute's breakdown of why warps weren't issuing instructions, such as Long Scoreboard (waiting for a memory load) or Math Pipe Throttle (math units busy).Source: Nsight Compute Kernel Profiling Guide (NVIDIA)Explained in Profiling and the Roofline: Finding What Limits a GPU Kernel
Wave quantization: Lost throughput when a grid's last wave of blocks fills only part of the GPU, leaving most SMs idle.Source: Matrix Multiplication Background (NVIDIA)Explained in The CUDA Programming Model: Threads, Warps, Blocks and SMs
Weight decay: Shrinking every weight slightly at each training step to keep weights small. AdamW applies it directly to the weights instead of through the gradient.Source: Decoupled Weight Decay Regularization (Loshchilov & Hutter, 2017)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Weight tying: Using the same matrix for a language model's input embedding table and its output layer, so one row per token does both jobs. It saves parameters and often improves predictions.Source: Using the Output Embedding to Improve Language Models (Press & Wolf, 2016)Explained in Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
Weight-stationary: A systolic-array dataflow in which each cell holds one weight while activations stream past it, so each weight tile is loaded once and reused for every input row.Source: In-Datacenter Performance Analysis of a Tensor Processing Unit (Jouppi et al., 2017)Explained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators
Welford's algorithm: A one-pass method that updates a running mean and sum of squared deviations, avoiding the cancellation of E[x²] − E[x]².Source: Note on a Method for Calculating Corrected Sums of Squares and Products (Welford, 1962)Explained in Reductions and Normalization Kernels: Summing Across Threads
wgmma: Hopper's asynchronous warpgroup MMA: four warps compute a tile of up to 64×256 per instruction, reading B (and optionally A) from shared memory.Source: PTX ISA (NVIDIA)Explained in Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
Wilson interval: A confidence interval for a proportion that stays sensible near 0% and 100%, unlike the simple ± formula. It treats every trial as independent.Source: Wilson (1927), Probable Inference, the Law of Succession, and Statistical InferenceExplained in Evaluating Agents: pass@k, pass^k and Run-to-Run Noise, Measured
WMMA: CUDA C++'s warp matrix API (nvcuda::wmma): load, multiply and store 16×16 tiles on tensor cores without knowing how elements map to threads.Source: CUDA C++ Programming Guide, Warp Matrix Functions (NVIDIA)Explained in Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
word2vec: A 2013 family of small, fast models from Google that learn one vector per word from large text collections. It made "king - man + woman ≈ queen" famous.Source: Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013)Explained in Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
WordPiece: Google's subword tokenization method, similar to BPE, used in GNMT and BERT.Source: Google's Neural Machine Translation System (Wu et al., 2016)
Write-ahead log: A record of an intended change saved before the change is made, so recovery after a crash can replay or finish it instead of repeating it.Source: Activity definition (Temporal docs)Explained in Long-Horizon Reliability: Compounding Errors, Retries and Checkpoints

X

Xavier initialization: Drawing a layer's starting weights with variance 2/(fan-in + fan-out), so signals keep roughly the same size going forward and gradients going backward.Source: Understanding the difficulty of training deep feedforward neural networks (Glorot & Bengio, 2010)Explained in Keeping Signals Stable: Initialization, Normalization and Residuals
XGrammar: A fast library for constrained decoding, used by serving engines such as vLLM and SGLang to enforce JSON schemas and grammars.Source: XGrammar (Dong et al., 2024)Explained in Guaranteeing Valid JSON Output from LLMs
XLA: Accelerated Linear Algebra: the compiler behind JAX and PyTorch/XLA, which compiles a whole computation for TPUs or GPUs and recompiles when tensor shapes change.Source: OpenXLA: XLAExplained in Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators

Y

YaRNYet another RoPE extensioN method: A method that stretches a RoPE model's context window by slowing only the slow-rotating dimensions and slightly sharpening attention, then fine-tuning briefly on long text.Source: YaRN: Efficient Context Window Extension of Large Language Models (Peng et al., 2023)Explained in Positional Encoding: How Transformers Know Word Order

Z

ZeRO / FSDP: Data-parallel training that splits optimizer state, gradients and weights across GPUs instead of copying them, so each GPU stores only a slice.Source: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., 2019)Explained in Pretraining: Data, Scaling Laws and Training at Scale
Zero point: An integer offset in asymmetric quantization that shifts the levels so they cover a range not centered on zero, such as -0.4 to 1.1.Source: Integer Quantization for Deep Learning Inference (Wu et al., 2020)Explained in Quantization: Running LLMs in 8 and 4 Bits