Fundamentals · Part 10How LLMs Work
The Modern LLM Block: What Changed Since 2017
RMSNorm, SwiGLU, no biases, grouped-query attention and RoPE: GPT-2's block next to Llama 3's and Qwen3's, with a full 8.03B parameter count.
Tagged: Normalization
2 articles.
Articles
Math · Part 10Math for LLMs
Keeping Signals Stable: Initialization, Normalization and Residuals
Why signals explode or vanish in deep networks, and the math of the fixes: Xavier and He init, LayerNorm and RMSNorm, residuals, pre-norm and μP.