Part 6
Cutting LLM Cost by 70% Without Losing Quality
How to stack prompt caching, model routing, batch APIs and token hygiene into a 70-80% cost cut, with an eval gate that proves quality held.
Cost, Latency & Serving
Make LLM systems cheaper and faster: caching, routing, batching and GPU serving.
Articles
Part 7
Reducing Time to First Token in a Chat App
Decompose TTFT into network, pre-model work, queueing and prefill, then cut each stage: parallelism, prefix caching, chunked prefill and headroom.
Part 8
Designing a Semantic Cache with Hit-Rate Tracking
A layered semantic cache for LLM responses: scoped keys, tuned thresholds, borderline verification, invalidation, and false-hit-aware hit-rate metrics.
Part 9
Routing Requests Between a Cheap and a Strong Model
Design an LLM router that sends each request to the cheapest good-enough model: rules, learned routers, cascades, threshold tuning and drift monitoring.
Part 21
Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization
How to size and run an open-weight LLM fleet: prefill vs decode, KV-cache math, continuous batching, FP8/FP4 quantization and goodput per dollar.