Inference & Optimization

Articles

• Inference · Part 1

  How LLM Inference Works: Prefill, Decode and the Memory Wall

  Why a GPU that can do a quadrillion operations a second spends most of its time waiting when it writes your answer, and what that means for speed and cost.

• Inference · Part 2

  The KV Cache: What It Stores, What It Costs, How to Shrink It

  What the KV cache holds, how big it gets on real models, and how MQA, GQA and DeepSeek's MLA shrink it so more users fit on one GPU.

• Inference · Part 3

  How GPUs Work for LLMs: Cores, Memory and Number Formats

  What is inside an H100: thousands of cores, tensor cores, a memory hierarchy from 256 KB to 80 GB, number formats down to FP4, and NVLink.

• Inference · Part 4

  Continuous Batching and Chunked Prefill: Keeping the GPU Full

  How LLM servers batch requests that start and finish at different times, and how chunked prefill stops one long prompt from freezing everyone's stream.

• Inference · Part 5

  PagedAttention and vLLM: Paging the KV Cache Like an OS

  Why early LLM servers wasted most of their KV cache memory, how PagedAttention's blocks and block tables fixed it, and how vLLM uses them today.

• Inference · Part 6

  Prefix Caching and KV Cache Offloading: Never Prefill Twice

  How servers keep KV caches between requests, find them again by hashing or a radix tree, and spill them to CPU memory and SSD when the GPU fills up.

• Inference · Part 7

  FlashAttention and Fused Kernels: Attention Without the n×n Matrix

  Why standard attention is memory-bound, how tiling and online softmax compute it exactly in on-chip SRAM, FlashAttention 1 to 4, and fused kernels.

• Inference · Part 8

  Quantization: Running LLMs in 8 and 4 Bits

  How LLMs run in 8 or 4 bits: scales and outliers, why decode speeds up, GPTQ, AWQ, SmoothQuant, FP8, FP4, KV cache quantization and checking quality.

• Inference · Part 9

  Speculative Decoding: Guess Several Tokens, Check Them in One Pass

  Why checking several guessed tokens costs about as much as writing one, how speculative sampling keeps the output exact, and Medusa, EAGLE and MTP.

• Inference · Part 10

  Tensor, Pipeline and Expert Parallelism: One Model, Many GPUs

  How to split a model too big for one GPU: tensor, pipeline, data and expert parallelism, what each costs in communication, and why NVLink decides the layout.

• Inference · Part 11

  Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction

  What 128K to 1M-token prompts cost on a GPU, and how sliding windows, sparse attention, KV cache eviction and hybrids cut attention and memory.

• Inference · Part 12

  Disaggregated Serving and llm-d: Splitting Prefill from Decode

  Why large LLM deployments run prefill and decode on separate GPUs, what moving the KV cache between them costs, and how llm-d runs it on Kubernetes.

• Inference · Part 13

  Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama

  How fast a model can run on your laptop and what fits: bandwidth math, unified memory, llama.cpp, GGUF quants, Ollama, MLX and measured numbers.

• Inference · Part 14

  Case Study: DeepSeek's Inference Stack and DeepInfer

  How DeepSeek serves its models so cheaply: MLA, FP8 kernels, wide expert parallelism, MTP, disaggregation and disk caching, and what DeepInfer releases.