Multimodal · Part 1Images, Speech & Video
How AI Sees, Hears and Draws: Everything Becomes Tokens
Images, audio and video all become sequences of vectors before a transformer sees them: a measured token ledger across eight open models and Claude.
Tagged: Vision
8 articles.
Articles
Multimodal · Part 2Images, Speech & Video
Vision Transformers: Patches, Positions and Pixels, Measured
How a vision transformer turns images into tokens, what its position embeddings and attention learn, and what resolution and data cost, measured on a laptop.
Multimodal · Part 3Images, Speech & Video
CLIP and Contrastive Learning: Training Images and Text Together
How CLIP learns from captions: the similarity matrix, InfoNCE, temperature, zero-shot vs probes, SigLIP, the modality gap and typographic attacks, measured.
Multimodal · Part 4Images, Speech & Video
Vision-Language Models: How an Image Becomes Tokens
Projectors, Q-Formers and cross-attention, tiling and pixel shuffle, what an image costs, and five checkable tasks run on SmolVLM, Qwen2-VL and Claude.
Multimodal · Part 5Images, Speech & Video
Diffusion From Scratch: Noise, Scores and a Laptop DDPM
The DDPM forward process, noise schedules, the noise-prediction loss and the score view, with a spiral and a Fashion-MNIST U-Net trained and measured.
Multimodal · Part 6Images, Speech & Video
Latent Diffusion: How Stable Diffusion Turns Text into Pictures
Stable Diffusion 1.5 taken apart and timed on a laptop GPU: the VAE, the U-Net, CLIP text, classifier-free guidance, negative prompts and samplers, measured.
Multimodal · Part 7Images, Speech & Video
Diffusion Transformers and Flow Matching: Straight Paths, Measured
How flow matching learns straight paths from noise to images, why reflow makes one step work, and what a DiT changes, measured on a laptop GPU.
Multimodal · Part 8Images, Speech & Video
Autoregressive Image Generation: Drawing Pictures as Tokens
VQ tokenizers, raster-order and masked transformers, next-scale prediction and unified models: how LLM-style models draw images, measured on a laptop.