Tagged: Cost

Articles

• Part 5Retrieval & RAG

  Adding a Reranker and Proving It Helped

  How to add a cross-encoder reranker to a RAG pipeline, size it, and prove the gain with paired offline evals, end-to-end checks and a guarded A/B test.

• Part 6Cost, Latency & Serving

  Cutting LLM Cost by 70% Without Losing Quality

  How to stack prompt caching, model routing, batch APIs and token hygiene into a 70-80% cost cut, with an eval gate that proves quality held.

• Part 8Cost, Latency & Serving

  Designing a Semantic Cache with Hit-Rate Tracking

  A layered semantic cache for LLM responses: scoped keys, tuned thresholds, borderline verification, invalidation, and false-hit-aware hit-rate metrics.

• Part 9Cost, Latency & Serving

  Routing Requests Between a Cheap and a Strong Model

  Design an LLM router that sends each request to the cheapest good-enough model: rules, learned routers, cascades, threshold tuning and drift monitoring.

• Part 19Fine-tuning & Adaptation

  Fine-Tuning vs RAG vs Prompting: An Eval-First Decision Procedure

  Diagnose whether your gap is knowledge or behavior, then climb from prompting to RAG to fine-tuning only when the cheaper rung plateaus.

• Part 21Cost, Latency & Serving

  Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization

  How to size and run an open-weight LLM fleet: prefill vs decode, KV-cache math, continuous batching, FP8/FP4 quantization and goodput per dollar.

• Part 25Production Engineering

  Designing a ChatGPT-Style Chat Service for 1M Daily Users

  Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.