Series
Beyond Text: Images, Speech and Video
How models see, hear and draw, measured on a laptop GPU: vision transformers, CLIP, vision-language models, diffusion and flow matching, Whisper, text-to-speech, voice agents and video generation. Read them in order, or jump to whatever you're curious about.
How AI Sees, Hears and Draws: Everything Becomes Tokens
Images, audio and video all become sequences of vectors before a transformer sees them: a measured token ledger across eight open models and Claude.
Images, Speech & VideoVision Transformers: Patches, Positions and Pixels, Measured
How a vision transformer turns images into tokens, what its position embeddings and attention learn, and what resolution and data cost, measured on a laptop.
Images, Speech & VideoCLIP and Contrastive Learning: Training Images and Text Together
How CLIP learns from captions: the similarity matrix, InfoNCE, temperature, zero-shot vs probes, SigLIP, the modality gap and typographic attacks, measured.
Images, Speech & VideoVision-Language Models: How an Image Becomes Tokens
Projectors, Q-Formers and cross-attention, tiling and pixel shuffle, what an image costs, and five checkable tasks run on SmolVLM, Qwen2-VL and Claude.
Images, Speech & VideoDiffusion From Scratch: Noise, Scores and a Laptop DDPM
The DDPM forward process, noise schedules, the noise-prediction loss and the score view, with a spiral and a Fashion-MNIST U-Net trained and measured.
Images, Speech & VideoLatent Diffusion: How Stable Diffusion Turns Text into Pictures
Stable Diffusion 1.5 taken apart and timed on a laptop GPU: the VAE, the U-Net, CLIP text, classifier-free guidance, negative prompts and samplers, measured.
Images, Speech & VideoDiffusion Transformers and Flow Matching: Straight Paths, Measured
How flow matching learns straight paths from noise to images, why reflow makes one step work, and what a DiT changes, measured on a laptop GPU.
Images, Speech & VideoAutoregressive Image Generation: Drawing Pictures as Tokens
VQ tokenizers, raster-order and masked transformers, next-scale prediction and unified models: how LLM-style models draw images, measured on a laptop.
Images, Speech & VideoSpeech Recognition with Whisper: From Waveform to Words
How Whisper turns sound into a log-mel spectrogram and then into text, and what word error rate, noise and silence do to it, measured on a laptop GPU.
Images, Speech & VideoText-to-Speech: How a Model Learns to Read Aloud
Text normalization, phonemes, duration, pitch and a waveform: how modern text-to-speech works, with Kokoro-82M timed and its intelligibility measured.
Images, Speech & VideoAudio Tokens and Voice Agents: Codecs, Turn-Taking and Latency
How neural codecs turn speech into tokens, how speech-to-speech models work, and where a voice agent's latency goes, measured stage by stage on a laptop.
Images, Speech & VideoVideo Generation: Spacetime Patches and Attention, Measured
How video models squeeze clips into spacetime patches, why attention cost explodes with clip length, and what temporal layers buy, measured on a laptop GPU.
Images, Speech & VideoServing Multimodal Models: Encoders, Image Tokens and Caching
Where the time and money go when a model reads images or audio: encoder cost, image-token prices, prompt and encoder caching, measured, plus a series review.
Images, Speech & Video