Series

LLMs on the GPU: Kernels and Hardware

How a GPU runs a language model, down to the kernels: CUDA, the memory hierarchy, tensor cores, Triton, attention and matmul kernels, and multi-GPU communication, measured on real hardware. Read them in order, or jump to whatever you're curious about.

  1. From torch.matmul to Silicon: What Happens When PyTorch Runs

    Follow one matmul from Python through the dispatcher, cuBLAS and a CUDA stream to the GPU, then trace one decode step of a real model on a laptop GPU.

    GPUs & Kernels

  2. The CUDA Programming Model: Threads, Warps, Blocks and SMs

    Write, compile and time a first CUDA kernel, see it hit the memory-bandwidth limit, and learn what warps, occupancy and grid size do to its speed.

    GPUs & Kernels

  3. The GPU Memory Hierarchy: Registers, Caches and DRAM, Measured

    Measure every level of a GPU's memory, from registers to DRAM, and see how access order, bank conflicts and spills cost most of the bandwidth.

    GPUs & Kernels

  4. Profiling and the Roofline: Finding What Limits a GPU Kernel

    Measure a laptop GPU's compute and bandwidth roofs, place real LLM kernels under them, and read Nsight and the PyTorch profiler to find each bottleneck.

    GPUs & Kernels

  5. Writing a Fast Matmul in CUDA, Step by Step

    Take an FP32 matrix multiply from a naive CUDA kernel to one that edges past cuBLAS on a laptop GPU, measuring and profiling every step.

    GPUs & Kernels

  6. Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured

    What one tensor-core instruction computes, how to feed it from shared memory without bank conflicts, and what FP16, BF16 and FP8 cost in speed and accuracy.

    GPUs & Kernels

  7. Triton: Writing GPU Kernels in Python, Measured

    Write vector add, fused softmax and a tensor-core matmul in Triton, autotune them, read the PTX they compile to, and time them against PyTorch.

    GPUs & Kernels

  8. Reductions and Normalization Kernels: Summing Across Threads

    Sum 67 million floats on a laptop GPU step by step, then write softmax, RMSNorm, LayerNorm and argmax kernels that stay fast and numerically stable.

    GPUs & Kernels

  9. Kernel Fusion, torch.compile and CUDA Graphs, Measured

    Fuse SwiGLU and RMSNorm by hand and with torch.compile, read the Triton it writes, replay decode as a CUDA graph, and see how PyTorch's allocator fragments.

    GPUs & Kernels

  10. Writing an Attention Kernel: FlashAttention and Decode in Triton

    Build a causal FlashAttention kernel and a split-KV, paged decode kernel in Triton, then time them against naive attention and PyTorch's SDPA.

    GPUs & Kernels

  11. Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured

    How 4-bit weight kernels dequantize in registers, when INT4, INT8 and FP8 make inference faster on a laptop GPU, and how FP8, MX and NVFP4 scales work.

    GPUs & Kernels

  12. Mixture-of-Experts Kernels: Routing, Grouped GEMM and Load Imbalance

    Route tokens, sort them by expert and run 128 experts in one grouped GEMM: a Triton MoE kernel timed against loops, padding and torch.grouped_mm.

    GPUs & Kernels

  13. Many GPUs: NVLink, NCCL and Collectives

    How GPUs add up tensors together: ring and tree all-reduce, the latency-bandwidth cost model, NVLink vs PCIe vs InfiniBand, and hiding the traffic.

    GPUs & Kernels

  14. Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators

    How AMD GPUs, Apple silicon, Google TPUs, AWS Trainium, Groq and Cerebras map onto the same ideas: threads, on-chip memory, matrix units, bandwidth and links.

    GPUs & Kernels

  15. One Token, Kernel by Kernel: A Decode Step on a Laptop GPU

    Every kernel in one Qwen2.5-1.5B decode step on a laptop RTX 4060: what it does, its time, its place on the roofline, and where the last 15% goes.

    GPUs & Kernels