Tagged: Caching

Articles

• Part 6Cost, Latency & Serving

  Cutting LLM Cost by 70% Without Losing Quality

  How to stack prompt caching, model routing, batch APIs and token hygiene into a 70-80% cost cut, with an eval gate that proves quality held.

• Part 7Cost, Latency & Serving

  Reducing Time to First Token in a Chat App

  Decompose TTFT into network, pre-model work, queueing and prefill, then cut each stage: parallelism, prefix caching, chunked prefill and headroom.

• Part 8Cost, Latency & Serving

  Designing a Semantic Cache with Hit-Rate Tracking

  A layered semantic cache for LLM responses: scoped keys, tuned thresholds, borderline verification, invalidation, and false-hit-aware hit-rate metrics.

• Part 21Cost, Latency & Serving

  Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization

  How to size and run an open-weight LLM fleet: prefill vs decode, KV-cache math, continuous batching, FP8/FP4 quantization and goodput per dollar.

• Part 25Production Engineering

  Designing a ChatGPT-Style Chat Service for 1M Daily Users

  Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.