Series
LLM Inference & Optimization
What happens when a model generates text, and the tricks that make it fast: KV caching, batching, PagedAttention, quantization, speculative decoding and more. Read them in order, or jump to whatever you're curious about.
How LLM Inference Works: Prefill, Decode and the Memory Wall
Why a GPU that can do a quadrillion operations a second spends most of its time waiting when it writes your answer, and what that means for speed and cost.
Inference & OptimizationThe KV Cache: What It Stores, What It Costs, How to Shrink It
What the KV cache holds, how big it gets on real models, and how MQA, GQA and DeepSeek's MLA shrink it so more users fit on one GPU.
Inference & OptimizationHow GPUs Work for LLMs: Cores, Memory and Number Formats
What is inside an H100: thousands of cores, tensor cores, a memory hierarchy from 256 KB to 80 GB, number formats down to FP4, and NVLink.
Inference & OptimizationContinuous Batching and Chunked Prefill: Keeping the GPU Full
How LLM servers batch requests that start and finish at different times, and how chunked prefill stops one long prompt from freezing everyone's stream.
Inference & OptimizationPagedAttention and vLLM: Paging the KV Cache Like an OS
Why early LLM servers wasted most of their KV cache memory, how PagedAttention's blocks and block tables fixed it, and how vLLM uses them today.
Inference & OptimizationPrefix Caching and KV Cache Offloading: Never Prefill Twice
How servers keep KV caches between requests, find them again by hashing or a radix tree, and spill them to CPU memory and SSD when the GPU fills up.
Inference & OptimizationFlashAttention and Fused Kernels: Attention Without the n×n Matrix
Why standard attention is memory-bound, how tiling and online softmax compute it exactly in on-chip SRAM, FlashAttention 1 to 4, and fused kernels.
Inference & OptimizationQuantization: Running LLMs in 8 and 4 Bits
How LLMs run in 8 or 4 bits: scales and outliers, why decode speeds up, GPTQ, AWQ, SmoothQuant, FP8, FP4, KV cache quantization and checking quality.
Inference & OptimizationSpeculative Decoding: Guess Several Tokens, Check Them in One Pass
Why checking several guessed tokens costs about as much as writing one, how speculative sampling keeps the output exact, and Medusa, EAGLE and MTP.
Inference & OptimizationTensor, Pipeline and Expert Parallelism: One Model, Many GPUs
How to split a model too big for one GPU: tensor, pipeline, data and expert parallelism, what each costs in communication, and why NVLink decides the layout.
Inference & OptimizationLong-Context Inference: Sliding Windows, Sparse Attention, KV Eviction
What 128K to 1M-token prompts cost on a GPU, and how sliding windows, sparse attention, KV cache eviction and hybrids cut attention and memory.
Inference & OptimizationDisaggregated Serving and llm-d: Splitting Prefill from Decode
Why large LLM deployments run prefill and decode on separate GPUs, what moving the KV cache between them costs, and how llm-d runs it on Kubernetes.
Inference & OptimizationRunning LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
How fast a model can run on your laptop and what fits: bandwidth math, unified memory, llama.cpp, GGUF quants, Ollama, MLX and measured numbers.
Inference & OptimizationCase Study: DeepSeek's Inference Stack and DeepInfer
How DeepSeek serves its models so cheaply: MLA, FP8 kernels, wide expert parallelism, MTP, disaggregation and disk caching, and what DeepInfer releases.
Inference & Optimization