Series

LLM System Design

Twenty-five deep dives on building production LLM systems, from retrieval and evaluation to serving, security and operations. Read them in order, or jump to whatever you're building.

  1. Designing a RAG System for 10 Million Documents

    How to size, build and measure retrieval-augmented generation over 10M documents: ingestion, hybrid retrieval, reranking, ACLs, cost and evals.

    Retrieval & RAG

  2. Debugging a RAG System That Gives Wrong Answers

    A stage-by-stage method for localizing RAG failures: build a replayable failure set, attribute each case to a stage, fix the biggest bucket, and gate on it.

    Retrieval & RAG

  3. Chunking Legal Contracts for Retrieval

    Why fixed-size chunks fail on contracts, and how to chunk by clause with context headers, defined terms, cross-references and amendment lineage.

    Retrieval & RAG

  4. BM25 vs Dense vs Hybrid Search: When to Use Each

    How lexical, dense, learned-sparse and late-interaction retrieval fail differently, how to fuse them with RRF, and how to choose on your own traffic.

    Retrieval & RAG

  5. Adding a Reranker and Proving It Helped

    How to add a cross-encoder reranker to a RAG pipeline, size it, and prove the gain with paired offline evals, end-to-end checks and a guarded A/B test.

    Retrieval & RAG

  6. Cutting LLM Cost by 70% Without Losing Quality

    How to stack prompt caching, model routing, batch APIs and token hygiene into a 70-80% cost cut, with an eval gate that proves quality held.

    Cost, Latency & Serving

  7. Reducing Time to First Token in a Chat App

    Decompose TTFT into network, pre-model work, queueing and prefill, then cut each stage: parallelism, prefix caching, chunked prefill and headroom.

    Cost, Latency & Serving

  8. Designing a Semantic Cache with Hit-Rate Tracking

    A layered semantic cache for LLM responses: scoped keys, tuned thresholds, borderline verification, invalidation, and false-hit-aware hit-rate metrics.

    Cost, Latency & Serving

  9. Routing Requests Between a Cheap and a Strong Model

    Design an LLM router that sends each request to the cheapest good-enough model: rules, learned routers, cascades, threshold tuning and drift monitoring.

    Cost, Latency & Serving

  10. Building Tool-Using Agents That Work Reliably

    Tool-use reliability is a systems property: tool design, a guarded control loop, permission gates, and pass^k evals on verifiable end states.

    Agents & Tool Use

  11. Detecting and Recovering from Agent Loops

    Classify why an agent loops, add harness guardrails and better tool feedback, and recover with an escalation ladder you can measure.

    Agents & Tool Use

  12. Designing Long-Term Memory for an LLM Assistant

    A tiered memory design with explicit write, read and forgetting policies, plus the benchmarks and production signals that show it works.

    Agents & Tool Use

  13. Guaranteeing Valid JSON Output from LLMs

    Enforce JSON Schema at decode time, design schemas that keep quality high, and add validation, repair and truncation handling on top.

    Agents & Tool Use

  14. Detecting and Reducing LLM Hallucinations

    Separate hallucination types, detect them with claim-level verification and uncertainty signals, and reduce them without making the model useless.

    Evaluation & Reliability

  15. Building an Eval Suite for a Customer Support Bot

    Build support-bot evals from real failures: layered offline tests, calibrated judges, statistically sound release gates and a production flywheel.

    Evaluation & Reliability

  16. Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift

    Treat an LLM judge like a measurement instrument: calibrate it against human labels, remove its known biases, and size your eval sets for real error bars.

    Evaluation & Reliability

  17. Defending Tool-Using Agents Against Prompt Injection by Design

    Why model-level filters can't stop prompt injection, and how control/data separation, taint tracking and sink policies make attacks harmless.

    Security & Privacy

  18. Building a PII Redaction Gateway in Front of Your LLM

    A layered design for detecting, pseudonymizing and re-identifying PII before it reaches a model, and how to measure the leak rate honestly.

    Security & Privacy

  19. Fine-Tuning vs RAG vs Prompting: An Eval-First Decision Procedure

    Diagnose whether your gap is knowledge or behavior, then climb from prompting to RAG to fine-tuning only when the cheaper rung plateaus.

    Fine-tuning & Adaptation

  20. Fine-Tuning a Small Model with LoRA on 5,000 Examples

    A practical LoRA recipe for an 8B model on 5K examples: data hygiene, hyperparameters, a real GPU memory budget, and eval-driven checkpoint selection.

    Fine-tuning & Adaptation

  21. Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization

    How to size and run an open-weight LLM fleet: prefill vs decode, KV-cache math, continuous batching, FP8/FP4 quantization and goodput per dollar.

    Cost, Latency & Serving

  22. Streaming LLM Responses over SSE and Surviving Disconnects

    Stream tokens with server-sent events, decouple generation from the connection, resume with Last-Event-ID, and cancel cleanly so GPUs are freed.

    Production Engineering

  23. Tracing Every Step of a Multi-Agent Workflow

    Model multi-agent runs as one OpenTelemetry trace: GenAI span conventions, context propagation across queues, private content storage and tail sampling.

    Production Engineering

  24. Rolling Back a Bad Prompt Change in Production

    Make prompt rollback a sub-minute alias flip, catch regressions with canaries and guardrail metrics, and clean up what the bad prompt left behind.

    Production Engineering

  25. Designing a ChatGPT-Style Chat Service for 1M Daily Users

    Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.

    Production Engineering