A ground-up Triton course, taught as self-contained HTML lessons. Written for someone with zero prior knowledge — mental models, analogies, interactive visualizations, and runnable companion scripts (Colab T4 GPU or local CPU interpreter).
Why GPUs exist, what a kernel is, the “write one, run many” shift, tiles, masks, constexpr — and your first vector add.
2D grids, broadcasting, strides, and a complete tiled matmul in ~25 lines — the exact kernel shape decode-lab’s GEMV and attention kernels are built from.
Block sums, the two-pass softmax dance, and decode-lab’s real fused RMSNorm kernel — the first kernel on the surgery map.
tl.make_block_ptr, tl.advance, and boundary checks — masks for free, and the
matmul rewritten in ~10 fewer lines.
One load, a register chain, one store. The fused SwiGLU MLP — decode-lab’s swiglu.py
pattern — and when not to fuse.
num_warps, num_stages, @triton.autotune, and honest benchmarking
with do_bench — plus the launch-bound truth about decode.
QKᵀ scores, causal mask, row softmax and PV in one kernel — with the online-softmax rescaling trick,
the core of decode-lab’s future attention.py.
Each lesson ships with a runnable Python companion — run it on Colab
(!pip install -q triton, T4 GPU) or locally via Triton’s CPU interpreter.
Old lesson versions are archived under artifacts/archive/.