Serving Multimodal Models: Encoders, Image Tokens and Caching

TL;DR

A request that carries an image or a sound clip has a stage a text request doesn’t: an encoder turns pixels or samples into vectors before the language model reads anything, and those vectors arrive as hundreds or thousands of input tokens. This finale measures where the time and money go: per-stage timings of two open vision-language models on a laptop GPU, what an image costs on hosted APIs, and two kinds of caching. Prompt caching of an image on Claude Haiku 5.5 cut the cost of each later question by up to 10 times, and reusing image features locally cut a follow-up turn from 1.22 s to 0.34 s with identical outputs. It closes with a review of all 13 parts.

Why it matters

Ask a 2-billion-parameter vision-language model on a laptop one question about a 1344 × 896 report page, and it spends 884 ms looking at the page before it reads a word of the question. That is 77% of the time to the first word of its answer. Ask a second question about the same page and a naive server looks at it all over again.

Hosted APIs charge for the same work. On Claude Haiku 5.5 one 1200 × 800 page is 1,250 input tokens, and with the question and answer that makes $0.13 per thousand questions. Mark the image for caching and every later question costs $0.018 per thousand, about 7 times less, from the second question on.

Inference Part 1 split a text request into prefill and decode. A multimodal request adds a third stage in front, and much of serving these models comes down to running that stage as rarely as possible.

The intuition

Think of a simultaneous interpreter handed a thick printed report before a press conference. Before they can answer any question about it, they have to read it, and reading takes longer than answering. A sensible interpreter reads it once and keeps notes. A careless one rereads the whole report every time a journalist raises a hand.

The vision encoder is the reading, and the notes are the encoder’s output, or further along, the language model’s KV cache (the stored keys and values of every earlier token, from Inference Part 2). Caching either one means the report is read once.

The analogy breaks in two places. A person’s notes stay useful if the report gets a new cover page; a model’s cached prefix becomes useless after any change, even a few pixels, because every later token depends on every earlier one. And a model can read many reports at once in a batch, which no interpreter can.

Step 1: what’s different about a multimodal request

Three things change compared with a text request.

1. An encoder stage runs before prefill. A vision transformer (ViT) for images, as in Part 2, or an audio encoder such as Whisper’s, as in Part 9. Its output passes through a connector into the language model’s input (Part 4).
2. Inputs are long. One picture becomes as many tokens as a few pages of text.
3. Outputs may not be text. A generator runs tens of denoising steps (Parts 5 to 7) or writes audio codec tokens that a decoder turns into sound (Part 11).

Earlier parts measured how many positions each input becomes:

Step 2: where the time goes in a local VLM request

The test input is a report page drawn in code: a bar chart and a small table of made-up depot figures, no people, logos or brands. Every question has an exact answer on the page.

Two open models ran it on the laptop’s NVIDIA GeForce RTX 4060 Laptop GPU (8 GB) in bf16, with PyTorch 2.11 and transformers 5.21.1: Qwen2-VL-2B-Instruct (a 675M-parameter ViT and a 1.5B language model) and SmolVLM-256M-Instruct, both Apache-2.0 and both from Part 4. Each request was timed in four stages: the processor (resizing, tiling and normalizing on the CPU, plus the copy to the GPU), the vision encoder with its connector, the language model’s prefill given the image features, and decode. Each number is the median of 7 runs after 2 warm-ups, with torch.cuda.synchronize() around each. Decode per token is the difference between generating 33 tokens and 1, divided by 32.

The transformers version used here accepts precomputed image features, which is what makes the stages separable:

with torch.inference_mode():
    feats = model.model.get_image_features(pixel_values=x["pixel_values"],
                                           image_grid_thw=x["image_grid_thw"], return_dict=True)
    text_only = {k: v for k, v in x.items() if k != "pixel_values"}
    out = model(**text_only, mm_encoder_outputs={"image": feats}, logits_to_keep=1)

Swipe to pan the diagram, or tap it to zoom.

The GPU stages of the two larger Qwen2-VL inputs stayed within 4% of their median; stages under 60 ms and the CPU processor varied more, up to 43% from the median. nvidia-smi read 1,545 to 2,160 MHz and 12 to 45 W, and peak GPU memory was 4.2 to 4.5 GB for Qwen2-VL and under 0.9 GB for SmolVLM.

Qwen2-VL’s encoder grows faster than its token count: 16 times the tokens cost 19 times the encoder time, because its ViT attends over every 14-pixel patch, 6,144 of them at 1344 × 896, before merging them 4 to 1. The encoder is half the time to first token for the small image and 77% for the large one, which matches Part 4’s 50% to 81% on square images.

SmolVLM’s slowest stage isn’t on the GPU at all. Its default processor enlarges the page to 2,048 pixels and cuts 12 tiles plus a global view, which took 301 ms of CPU time, 60% of its time to first token. That is why vLLM caches processor output too; its design notes say “some HF processors, such as the one for Qwen2-VL, are very slow.”

Decode costs 24 to 27 ms per token whatever the image, on both models. Once the image is in the KV cache it’s just more context, and each step is dominated by per-step overhead and weight reads, as Inference Part 1 explained.

A first attempt took 7.9 s for the large encoder and 15.7 GB of peak memory, twice the card: the features had been computed outside torch.inference_mode(), so PyTorch kept every activation for a backward pass that never came. Serving code needs gradients off.

Step 3: what an image costs on a hosted API

Claude’s rule, extended

Anthropic’s vision docs give the rule: “Each patch is a 28×28-pixel block of the image, referred to as a visual token. An image, therefore, costs ⌈width / 28⌉ × ⌈height / 28⌉ visual tokens.” Claude 4.7 and later models, Haiku 5.5 among them, accept up to 2,576 pixels on the long edge and 4,784 visual tokens; larger images are shrunk to fit. Parts 1 and 4 checked that against the free token-counting endpoint at 16 sizes and found the formula plus 3 every time.

Three more checks, on claude-haiku-5-5, subtracting the count for the question alone:

The 3 extra tokens come with every image, not once per request: 8 pages cost 8 × (1,247 + 3). Compression changes the upload size fourfold and the bill not at all, since Claude counts patches, not bytes. A PDF costs more than its page as a picture. The PDF docs explain: “The system converts each page of the document into an image,” and “the text from each page is extracted and provided alongside each page’s image.” The image-only PDF cost the same 1,617 tokens at two page sizes, so the API renders PDF pages at its own resolution, and the text layer added 165 tokens, about what the page’s text costs as plain text (166).

Other providers

Each provider tokenizes media differently, read from their own docs on November 5, 2026, and not measured here:

The same 1024 × 1024 image is 1,369 tokens on Claude, 1,229 on gpt-6-astra at high detail and, by Google’s rule, 4 tiles of 258 = 1,032 on Gemini, if each tile is a full 768 × 768 square. Token counts aren’t prices: multiply each by that model’s own input price.

Step 4: prompt caching with images

Prompt caching, covered for text in Inference Part 6, keeps the server’s KV cache for a marked prefix so a later request that starts with the same bytes skips that part of prefill. Claude’s prompt-caching docs list “Images and documents” in user turns among what can be cached, as long as the prompt is “100 percent identical … including all text and images up to and including the block marked with cache control.” Haiku 5.5’s minimum is 512 tokens, so one 1200 × 800 image clears it alone.

The pricing page sets the trade for Haiku 5.5 (prompts up to 100,000 tokens): $0.10 per million input tokens, $0.125 per million for 5-minute cache writes (1.25 times), and $0.01 for cache hits (0.1 times).

The experiment

Three arms ran interleaved in rounds, in shuffled order, with a new question each round shared by all three. The image went first and the question after it, as the vision docs recommend:

• plain: the image with no cache_control.
• write: cache_control on the image, but the page footer carries a unique reference string, so every call is a cache miss that writes a new entry.
• hit: cache_control on the identical image, written once by a priming call before the first round.

Every call used claude-haiku-5-5 with thinking disabled, max_tokens 40 and a one-line system prompt asking for the number only, streamed through the Anthropic SDK; time to first token (TTFT) is the first text delta. Three sizes: S, page 1 at 1200 × 800; L, page 1 at 2352 × 1568; XL, all eight pages at 2352 × 1568. That made 40 rounds per size and 360 calls, on November 5, 2026, from India over the public API.

blocks = [image_block(page) for page in pages]          # base64 PNGs
blocks[-1]["cache_control"] = {"type": "ephemeral"}      # cache everything up to the last image
with client.messages.stream(model="claude-haiku-5-5", max_tokens=40, system=SYSTEM,
                            thinking={"type": "disabled"},
                            messages=[{"role": "user", "content": blocks + [{"type": "text", "text": q}]}]) as s:
    ...
print(s.get_final_message().usage)   # input_tokens, cache_creation_input_tokens, cache_read_input_tokens

Tokens are the median usage per call, costs the mean from the usage ledger, intervals bootstrapped over the 40 calls; p90 is the time 1 call in 10 exceeded.

Every call reported the split it should have: plain never read the cache, and each write stored the system prompt plus the images, leaving only the question’s 27 tokens as ordinary input. XL’s priming call read 4,731 tokens from the cache, the system prompt and page 1 that the L run had left there minutes before, because they were an exact prefix of the XL prompt. The last number in each reply matched the page in 38 to 40 of the 40 calls in every arm, no reply hit the 40-token limit, and all three arms gave the same answer in 38, 38 and 40 of the 40 rounds.

When caching pays

Write T for the cached prefix in tokens and n for the number of questions. Without caching each question pays T at the base price. With caching the first pays 1.25 T and each later one 0.1 T:

no cache:  n × T × $0.10 per million
cache:     (1.25 + 0.1 × (n - 1)) × T × $0.10 per million
n = 1:  1.00 T  vs 1.25 T   caching costs 25% more
n = 2:  2.00 T  vs 1.35 T   caching is already cheaper
n = 10: 10.0 T  vs 2.15 T   4.7 times cheaper

With the measured usage, question and output tokens included, the S page cost $0.000133 per question uncached, $0.000165 for the write and $0.000018 per hit: cheaper from the second question, and ten questions for $0.00032 instead of $0.0013. For the eight-page XL prompt, a question cost $0.00377 uncached, $0.00471 for the write and $0.00038 per hit, so ten questions came to $0.0081 instead of $0.038. The one condition is time: a 5-minute entry expires unless a hit refreshes it, so questions spaced more than 5 minutes apart need the 1-hour write, at 2 times the base price, which the docs say “pays off after two cache reads.”

Swipe to pan the diagram, or tap it to zoom.

Speed moved much less than cost. Paired within each round, a hit beat the uncached call by a median 0.08 s on S (95% CI 0.02 to 0.17 s; faster in 31 of 40 rounds), by nothing on L (0.08 s slower, interval −0.04 to +0.16), and by 0.25 s on XL (0.13 to 0.54 s; 30 of 40). A cache write added a median 0.12 to 0.23 s over plain, though only for L did the 95% interval exclude zero. Every arm still uploads its images, 2.5 MB of base64 for XL, and a text-only Haiku call from the same place took 0.78 s to its first token in Part 11, so most of the wait is network and queueing that no cache removes. Agentic Part 15 saw the same on agents: caching cut cost by 37% and left speed unchanged. Here the time saved only showed at 37,680 cached tokens.

Step 5: caching and batching in a local server

Reusing image features across turns

On the laptop, a 6-turn conversation about the same 1344 × 896 page (1,536 image tokens) with Qwen2-VL-2B, greedy decoding, ran three ways, 1 warm-up conversation and 5 timed:

• A, stateless: every turn runs the processor and encoder again and prefills the whole conversation.
• B, encoder cache: turn 1 stores the image features; later turns only tokenize the text and pass the stored features (mm_encoder_outputs).
• C, encoder and prefix cache: B plus the KV cache of earlier turns, trimmed to the longest common prefix, so each turn prefills only its new tokens.

Every turn of every timed run fell within 6% of its median except one C turn at 307 ms; nvidia-smi read a median 2,010 MHz and 44 W. All 6 answers were right in every variant.

Caching the image features cut a follow-up turn 3.6 times, because the 884 ms encoder ran once instead of six times. Reusing the KV cache cut it to 217 ms, about 100 ms of it the 3 or 4 decoded tokens at 25 ms each. B’s first-token logits matched A’s exactly (largest difference 0.0); C’s differed by at most 0.31 on logits of size 27, the rounding of bf16 kernels run on different shapes, and no reply changed.

What the serving systems do

vLLM and SGLang, the open servers from Inference Parts 5 and 6, build these steps in. Described from their documentation, not run here:

• Processor cache. vLLM’s multimodal processor cache “is automatically enabled to avoid repeatedly processing the same multi-modal inputs”, sized by mm_processor_cache_gb (default 4 GiB).
• Images in the prefix cache. vLLM hashes each 16-token KV block with its tokens, but image placeholder tokens look the same whatever the picture, so “we encode the image hash generated by the frontend image processor” into each block’s hash. Two requests share cached blocks only if they carry the same image.
• Encoder disaggregation. vLLM’s disaggregated encoder “runs the vision-encoder stage of a multimodal LLM in a process that is separate from the pre-fill / decoder stage”, which also allows “cross-process reuse and caching of encoder outputs.” SGLang’s EPD (encoder, prefill, decode) disaggregation launches --encoder-only and --language-only servers, with an optional global cache of ViT embeddings shared across instances. The EPD paper (Singh et al., 2025) reports “up to 15x lower peak memory utilization”, “up to 22x larger” batches and “up to 71%” lower TTFT against systems that don’t split the stages (reported, not reproduced here).
• Encoder parallelism. Vision encoders are small, so splitting their weights across GPUs buys little. vLLM’s mm_encoder_tp_mode="data" splits the batch of images instead, which its docs say improved “throughput and TTFT by around 10% for tensor_parallel_size=8”; SGLang’s --mm-enable-dp-encoder does the same.

Batching the encoder doesn’t help here

The same Qwen2-VL-2B served batches of distinct 672 × 448 pages (384 image tokens each), median of 5 runs after warm-up:

The encoder gives 6 images per second at any batch size: one 1,536-patch image already keeps the GPU’s arithmetic busy, and this transformers version runs attention one image at a time. Decode is what batching speeds up, since all 16 requests share each read of the weights, so whole requests rose 2.4 times while the encoder stayed flat. At batch 32 peak memory passed what the 8 GB card had free and throughput fell. That asymmetry, a compute-bound encoder next to a memory-bound decoder, is the case for running them on separate workers, as disaggregated prefill and decode does for text.

Step 6: serving generators and audio

A generator’s cost is roughly steps × cost per step; the cost per step grows with resolution, and classifier-free guidance doubles the work. Earlier parts measured each knob:

• Steps. Stable Diffusion 1.5 took 1.09 s per 512-pixel image at 8 steps, 3.5 s at 30 and 11.6 s at 100, in batches of 4, and 8 steps already matched the 100-step image’s CLIP score (Part 6).
• Guidance. Running the U-Net with and without the prompt cost 128.7 ms per step against 80.2 ms for one pass, 1.6 times (Part 6).
• Resolution. A 5-second 720p clip is 75,600 tokens, and one full attention call over them took 491 ms on the laptop against 32 ms factorized (Part 12).

Few-step distillation attacks the first knob. A student model is trained to jump where the teacher needs many steps: Latent Consistency Models (Luo et al., 2023) report “2~4-step” sampling trained in “32 A100 GPU hours”, and Adversarial Diffusion Distillation (Sauer et al., 2023) “in just 1-4 steps” (both reported, not reproduced here). Part 7 showed the same idea at toy scale: one round of reflow took a one-step sample’s error from 724 to 46.5.

For a server, generators are the opposite of VLMs: no long prompt to cache, and a full network pass at every step, so batching requests per step and cutting steps matter most.

For audio, streaming changes the budget more than any model choice. Part 11’s cascaded voice agent took 2.41 s from the end of speech to its first sound, and 2.02 s when the first finished sentence went straight to synthesis; Part 10 found sentence-by-sentence synthesis brought Kokoro’s first audio from 325 ms to 72 ms. Chunk sizes set the floor: 32 ms detector chunks, a 500 ms silence rule that fired 605 ms after the last word, Mimi’s 80 ms codec frame.

Whisper pads every clip to a 30-second window (Part 9), so it doesn’t stream by design. Whisper-Streaming (Macháček et al., 2023) re-transcribes a growing buffer, commits only words that consecutive passes agree on, and reports “3.3 seconds latency” on long-form speech (reported, not reproduced here). Speech-to-speech models like Moshi skip the chain, at “200ms in practice.”

The series in one table

Each part’s headline measurement, all on the same RTX 4060 Laptop GPU or Claude’s API:

What this series didn’t cover:

• Fine-tuning vision-language and speech models, for example with LoRA on your own documents or accents.
• Image editing, inpainting and control signals such as ControlNet, and music generation.
• 3D and scene representations (NeRFs, Gaussian splatting) and vision-language-action models for robots.
• Long video understanding, where frame sampling and token budgets decide what a model can see.
• Provenance and misuse: watermarking, content credentials and detecting generated media.
• Large-scale multimodal benchmarks; the experiments here used synthetic, checkable task sets.

In the real world

• vLLM’s disaggregated-encoder example runs Qwen/Qwen2.5-VL-3B-Instruct as one encoder instance feeding one prefill-decode instance, and its proxy passes audio for models such as Qwen2-Audio and Qwen3-Omni as “metadata-only references”, so the second instance never repeats audio preprocessing.
• SGLang’s EPD example puts two encoder servers in front of one language server for Qwen3-VL-8B, for workloads where the encoder is the slow stage.
• Claude’s vision docs say high resolution “can use up to roughly three times more visual tokens” and advise downsampling when the detail isn’t needed.
• At Gemini’s 32 tokens per second, an hour of audio is 115,200 tokens before any answer is written.

Common misconceptions

• “A smaller image file is a cheaper image.” Claude counts 28-pixel patches. PNG, JPEG and WebP versions of the same page, from 105 KB down to 28 KB, all cost 1,250 tokens.
• “Images make every token slower.” They make the first token slower. Qwen2-VL decoded at 24 to 27 ms per token whether the image was 96 or 1,536 tokens.
• “The GPU is always the bottleneck.” SmolVLM’s default processor spent 301 ms on the CPU, more than its encoder and prefill together.
• “Caching changes what the model answers.” Reused image features gave bit-identical logits, a reused KV cache moved them by bf16 rounding, and no reply changed in 30 turns.
• “Caching only pays for long conversations.” On Claude a cached image is cheaper from the second question, because a hit costs a tenth of the base price.

Key takeaways

• A multimodal request adds an encoder stage before prefill. On a laptop it was half of Qwen2-VL’s time to first token for a small image and 77% for a large one; decode didn’t notice the image.
• Image tokens follow each provider’s own rule: 28-pixel patches on Claude, 32-pixel patches times a model multiplier on OpenAI’s newer models, 768-pixel tiles of 258 tokens on Gemini. File size doesn’t change the count.
• Prompt caching an image on Claude Haiku 5.5 costs 25% more on the first question and 90% less on every later one, so it pays from question two.
• Locally, keep the image features and the KV cache: a follow-up turn fell from 1.22 s to 0.34 s and then 0.22 s, with the same answers.
• Batching helps decode, not a busy encoder; that’s why servers can now run the encoder on its own workers. Generators and voice agents are served by cutting steps and streaming chunks.

What’s next

This was the last part of Beyond Text. To go deeper on the serving machinery used here, the Inference series covers batching, paging, caching and disaggregation for text models. Measuring LLMs shows how to test models properly, and Agentic AI puts models like these to work in loops with tools, screenshots and voices.

Further reading

• Vision (Claude API docs): the visual-token formula, resolution tiers, limits and image placement.
• Prompt caching (Claude API docs): what can be cached, minimum lengths, breakpoints and invalidation.
• Pricing (Claude API docs): base, cache write, cache hit and batch prices for Haiku 5.5.
• PDF support (Claude API docs): how pages are converted into an image plus extracted text.
• Efficiently Serving Large Multimodal Models Using EPD Disaggregation (Singh et al., 2025): separating encode, prefill and decode onto different resources.
• vLLM: disaggregated encoder, optimization and multimodal caching and prefix caching design: processor caching, image hashes in KV blocks and encoder data parallelism.
• SGLang: EPD disaggregation and DP for the multimodal encoder.
• Latent Consistency Models (Luo et al., 2023) and Adversarial Diffusion Distillation (Sauer et al., 2023): few-step image generation by distillation.
• Turning Whisper into Real-Time Transcription System (Macháček et al., 2023): streaming recognition on top of a 30-second-window model.
• Images and vision (OpenAI docs) and Understand and count tokens (Gemini API docs): other providers’ image and audio token rules.
• Qwen2-VL-2B-Instruct and SmolVLM-256M-Instruct (Hugging Face model cards): the two local models, both Apache-2.0.
• Prefix Caching and KV Cache Offloading (Inference Part 6) and Cost and Latency of Agents (Agentic Part 15): the text-only versions of the caching measured here.