GPUs & Kernels

Articles

• GPU · Part 1

  From torch.matmul to Silicon: What Happens When PyTorch Runs

  Follow one matmul from Python through the dispatcher, cuBLAS and a CUDA stream to the GPU, then trace one decode step of a real model on a laptop GPU.

• GPU · Part 2

  The CUDA Programming Model: Threads, Warps, Blocks and SMs

  Write, compile and time a first CUDA kernel, see it hit the memory-bandwidth limit, and learn what warps, occupancy and grid size do to its speed.

• GPU · Part 3

  The GPU Memory Hierarchy: Registers, Caches and DRAM, Measured

  Measure every level of a GPU's memory, from registers to DRAM, and see how access order, bank conflicts and spills cost most of the bandwidth.

• GPU · Part 4

  Profiling and the Roofline: Finding What Limits a GPU Kernel

  Measure a laptop GPU's compute and bandwidth roofs, place real LLM kernels under them, and read Nsight and the PyTorch profiler to find each bottleneck.

• GPU · Part 5

  Writing a Fast Matmul in CUDA, Step by Step

  Take an FP32 matrix multiply from a naive CUDA kernel to one that edges past cuBLAS on a laptop GPU, measuring and profiling every step.

• GPU · Part 6

  Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured

  What one tensor-core instruction computes, how to feed it from shared memory without bank conflicts, and what FP16, BF16 and FP8 cost in speed and accuracy.

• GPU · Part 7

  Triton: Writing GPU Kernels in Python, Measured

  Write vector add, fused softmax and a tensor-core matmul in Triton, autotune them, read the PTX they compile to, and time them against PyTorch.

• GPU · Part 8

  Reductions and Normalization Kernels: Summing Across Threads

  Sum 67 million floats on a laptop GPU step by step, then write softmax, RMSNorm, LayerNorm and argmax kernels that stay fast and numerically stable.

• GPU · Part 9

  Kernel Fusion, torch.compile and CUDA Graphs, Measured

  Fuse SwiGLU and RMSNorm by hand and with torch.compile, read the Triton it writes, replay decode as a CUDA graph, and see how PyTorch's allocator fragments.

• GPU · Part 10

  Writing an Attention Kernel: FlashAttention and Decode in Triton

  Build a causal FlashAttention kernel and a split-KV, paged decode kernel in Triton, then time them against naive attention and PyTorch's SDPA.

• GPU · Part 11

  Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured

  How 4-bit weight kernels dequantize in registers, when INT4, INT8 and FP8 make inference faster on a laptop GPU, and how FP8, MX and NVFP4 scales work.

• GPU · Part 12

  Mixture-of-Experts Kernels: Routing, Grouped GEMM and Load Imbalance

  Route tokens, sort them by expert and run 128 experts in one grouped GEMM: a Triton MoE kernel timed against loops, padding and torch.grouped_mm.

• GPU · Part 13

  Many GPUs: NVLink, NCCL and Collectives

  How GPUs add up tensors together: ring and tree all-reduce, the latency-bandwidth cost model, NVLink vs PCIe vs InfiniBand, and hiding the traffic.

• GPU · Part 14

  Beyond NVIDIA: AMD, Apple, TPUs and Other AI Accelerators

  How AMD GPUs, Apple silicon, Google TPUs, AWS Trainium, Groq and Cerebras map onto the same ideas: threads, on-chip memory, matrix units, bandwidth and links.

• GPU · Part 15

  One Token, Kernel by Kernel: A Decode Step on a Laptop GPU

  Every kernel in one Qwen2.5-1.5B decode step on a laptop RTX 4060: what it does, its time, its place on the roofline, and where the last 15% goes.