Multimodal · Part 1
How AI Sees, Hears and Draws: Everything Becomes Tokens
Images, audio and video all become sequences of vectors before a transformer sees them: a measured token ledger across eight open models and Claude.
Images, Speech & Video
Vision transformers, CLIP, vision-language models, diffusion, speech recognition, text-to-speech and video: how models work beyond text.
Articles
Multimodal · Part 2
Vision Transformers: Patches, Positions and Pixels, Measured
How a vision transformer turns images into tokens, what its position embeddings and attention learn, and what resolution and data cost, measured on a laptop.
Multimodal · Part 3
CLIP and Contrastive Learning: Training Images and Text Together
How CLIP learns from captions: the similarity matrix, InfoNCE, temperature, zero-shot vs probes, SigLIP, the modality gap and typographic attacks, measured.
Multimodal · Part 4
Vision-Language Models: How an Image Becomes Tokens
Projectors, Q-Formers and cross-attention, tiling and pixel shuffle, what an image costs, and five checkable tasks run on SmolVLM, Qwen2-VL and Claude.
Multimodal · Part 5
Diffusion From Scratch: Noise, Scores and a Laptop DDPM
The DDPM forward process, noise schedules, the noise-prediction loss and the score view, with a spiral and a Fashion-MNIST U-Net trained and measured.
Multimodal · Part 6
Latent Diffusion: How Stable Diffusion Turns Text into Pictures
Stable Diffusion 1.5 taken apart and timed on a laptop GPU: the VAE, the U-Net, CLIP text, classifier-free guidance, negative prompts and samplers, measured.
Multimodal · Part 7
Diffusion Transformers and Flow Matching: Straight Paths, Measured
How flow matching learns straight paths from noise to images, why reflow makes one step work, and what a DiT changes, measured on a laptop GPU.
Multimodal · Part 8
Autoregressive Image Generation: Drawing Pictures as Tokens
VQ tokenizers, raster-order and masked transformers, next-scale prediction and unified models: how LLM-style models draw images, measured on a laptop.
Multimodal · Part 9
Speech Recognition with Whisper: From Waveform to Words
How Whisper turns sound into a log-mel spectrogram and then into text, and what word error rate, noise and silence do to it, measured on a laptop GPU.
Multimodal · Part 10
Text-to-Speech: How a Model Learns to Read Aloud
Text normalization, phonemes, duration, pitch and a waveform: how modern text-to-speech works, with Kokoro-82M timed and its intelligibility measured.
Multimodal · Part 11
Audio Tokens and Voice Agents: Codecs, Turn-Taking and Latency
How neural codecs turn speech into tokens, how speech-to-speech models work, and where a voice agent's latency goes, measured stage by stage on a laptop.
Multimodal · Part 12
Video Generation: Spacetime Patches and Attention, Measured
How video models squeeze clips into spacetime patches, why attention cost explodes with clip length, and what temporal layers buy, measured on a laptop GPU.
Multimodal · Part 13
Serving Multimodal Models: Encoders, Image Tokens and Caching
Where the time and money go when a model reads images or audio: encoder cost, image-token prices, prompt and encoder caching, measured, plus a series review.