Tagged: Triton

Articles

• GPU · Part 7GPUs & Kernels

  Triton: Writing GPU Kernels in Python, Measured

  Write vector add, fused softmax and a tensor-core matmul in Triton, autotune them, read the PTX they compile to, and time them against PyTorch.

• GPU · Part 9GPUs & Kernels

  Kernel Fusion, torch.compile and CUDA Graphs, Measured

  Fuse SwiGLU and RMSNorm by hand and with torch.compile, read the Triton it writes, replay decode as a CUDA graph, and see how PyTorch's allocator fragments.

• GPU · Part 10GPUs & Kernels

  Writing an Attention Kernel: FlashAttention and Decode in Triton

  Build a causal FlashAttention kernel and a split-KV, paged decode kernel in Triton, then time them against naive attention and PyTorch's SDPA.

• GPU · Part 11GPUs & Kernels

  Low-Precision Kernels: INT4, INT8 and FP8 Matmuls, Measured

  How 4-bit weight kernels dequantize in registers, when INT4, INT8 and FP8 make inference faster on a laptop GPU, and how FP8, MX and NVFP4 scales work.

• GPU · Part 12GPUs & Kernels

  Mixture-of-Experts Kernels: Routing, Grouped GEMM and Load Imbalance

  Route tokens, sort them by expert and run 128 experts in one grouped GEMM: a Triton MoE kernel timed against loops, padding and torch.grouped_mm.