3 articles.
Fundamentals · Part 11How LLMs Work
How MoE models work: a router picks a few expert MLPs per token, why total and active parameters differ, and how training keeps experts balanced.
Math · Part 15Math for LLMs
Exact parameter counts from a config file, 6ND derived with what it leaves out, KV cache bytes, log-log fits, the Chinchilla optimum and MoE routing math.
GPU · Part 12GPUs & Kernels
Route tokens, sort them by expert and run 128 experts in one grouped GEMM: a Triton MoE kernel timed against loops, padding and torch.grouped_mm.
