Math · Part 14Math for LLMs
Floating Point and Quantization: The Math of Fewer Bits
How computers store decimals, why 256 + 1 = 256 in BF16, why training keeps FP32 copies, and the error math of scales, groups, GPTQ and AWQ on real weights.
Tagged: Quantization
6 articles.
Articles
From Scratch · Part 15Training LLMs
Build an LLM From Scratch: From Our Model to Frontier Models
Our 88M run next to Llama 3, DeepSeek-V3, OLMo 2 and SmolLM, the model exported to Hugging Face and llama.cpp, and the whole series reviewed.
Inference · Part 8Inference & Optimization
Quantization: Running LLMs in 8 and 4 Bits
How LLMs run in 8 or 4 bits: scales and outliers, why decode speeds up, GPTQ, AWQ, SmoothQuant, FP8, FP4, KV cache quantization and checking quality.
Inference · Part 13Inference & Optimization
Running LLMs on Your Own Machine: llama.cpp, GGUF and Ollama
How fast a model can run on your laptop and what fits: bandwidth math, unified memory, llama.cpp, GGUF quants, Ollama, MLX and measured numbers.
GPU · Part 6GPUs & Kernels
Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured
What one tensor-core instruction computes, how to feed it from shared memory without bank conflicts, and what FP16, BF16 and FP8 cost in speed and accuracy.
GPU · Part 11GPUs & Kernels
Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured
How 4-bit weight kernels dequantize in registers, when INT4, INT8 and FP8 make inference faster on a laptop GPU, and how FP8, MX and NVFP4 scales work.