Tagged: Cuda

Articles

• GPU · Part 1GPUs & Kernels

  From torch.matmul to Silicon: What Happens When PyTorch Runs

  Follow one matmul from Python through the dispatcher, cuBLAS and a CUDA stream to the GPU, then trace one decode step of a real model on a laptop GPU.

• GPU · Part 2GPUs & Kernels

  The CUDA Programming Model: Threads, Warps, Blocks and SMs

  Write, compile and time a first CUDA kernel, see it hit the memory-bandwidth limit, and learn what warps, occupancy and grid size do to its speed.

• GPU · Part 3GPUs & Kernels

  The GPU Memory Hierarchy: Registers, Caches and DRAM, Measured

  Measure every level of a GPU's memory, from registers to DRAM, and see how access order, bank conflicts and spills cost most of the bandwidth.

• GPU · Part 4GPUs & Kernels

  Profiling and the Roofline: Finding What Limits a GPU Kernel

  Measure a laptop GPU's compute and bandwidth roofs, place real LLM kernels under them, and read Nsight and the PyTorch profiler to find each bottleneck.

• GPU · Part 5GPUs & Kernels

  Writing a Fast Matmul in CUDA, Step by Step

  Take an FP32 matrix multiply from a naive CUDA kernel to one that edges past cuBLAS on a laptop GPU, measuring and profiling every step.

• GPU · Part 6GPUs & Kernels

  Tensor Cores: Matrix Instructions, Swizzling and FP8, Measured

  What one tensor-core instruction computes, how to feed it from shared memory without bank conflicts, and what FP16, BF16 and FP8 cost in speed and accuracy.

• GPU · Part 7GPUs & Kernels

  Triton: Writing GPU Kernels in Python, Measured

  Write vector add, fused softmax and a tensor-core matmul in Triton, autotune them, read the PTX they compile to, and time them against PyTorch.

• GPU · Part 8GPUs & Kernels

  Reductions and Normalization Kernels: Summing Across Threads

  Sum 67 million floats on a laptop GPU step by step, then write softmax, RMSNorm, LayerNorm and argmax kernels that stay fast and numerically stable.

• GPU · Part 9GPUs & Kernels

  Kernel Fusion, torch.compile and CUDA Graphs, Measured

  Fuse SwiGLU and RMSNorm by hand and with torch.compile, read the Triton it writes, replay decode as a CUDA graph, and see how PyTorch's allocator fragments.

• GPU · Part 11GPUs & Kernels

  Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured

  How 4-bit weight kernels dequantize in registers, when INT4, INT8 and FP8 make inference faster on a laptop GPU, and how FP8, MX and NVFP4 scales work.

• GPU · Part 13GPUs & Kernels

  Many GPUs: NVLink, NCCL and Collectives

  How GPUs add up tensors together: ring and tree all-reduce, the latency-bandwidth cost model, NVLink vs PCIe vs InfiniBand, and hiding the traffic.

• GPU · Part 15GPUs & Kernels

  One Token, Kernel by Kernel: A Decode Step on a Laptop GPU

  Every kernel in one Qwen2.5-1.5B decode step on a laptop RTX 4060: what it does, its time, its place on the roofline, and where the last 15% goes.