Tagged: Routing

Articles

• Part 6Cost, Latency & Serving

  Cutting LLM Cost by 70% Without Losing Quality

  How to stack prompt caching, model routing, batch APIs and token hygiene into a 70-80% cost cut, with an eval gate that proves quality held.

• Part 9Cost, Latency & Serving

  Routing Requests Between a Cheap and a Strong Model

  Design an LLM router that sends each request to the cheapest good-enough model: rules, learned routers, cascades, threshold tuning and drift monitoring.

• Part 25Production Engineering

  Designing a ChatGPT-Style Chat Service for 1M Daily Users

  Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.