All articles

25 deep dives on designing production LLM systems, grouped by topic.

Retrieval & RAG

Chunking, hybrid search, reranking and debugging retrieval-augmented generation.

Part 1
Designing a RAG System for 10 Million Documents
How to size, build and measure retrieval-augmented generation over 10M documents: ingestion, hybrid retrieval, reranking, ACLs, cost and evals.
Part 2
Debugging a RAG System That Gives Wrong Answers
A stage-by-stage method for localizing RAG failures: build a replayable failure set, attribute each case to a stage, fix the biggest bucket, and gate on it.
Part 3
Chunking Legal Contracts for Retrieval
Why fixed-size chunks fail on contracts, and how to chunk by clause with context headers, defined terms, cross-references and amendment lineage.
Part 4
BM25 vs Dense vs Hybrid Search: When to Use Each
How lexical, dense, learned-sparse and late-interaction retrieval fail differently, how to fuse them with RRF, and how to choose on your own traffic.
Part 5
Adding a Reranker and Proving It Helped
How to add a cross-encoder reranker to a RAG pipeline, size it, and prove the gain with paired offline evals, end-to-end checks and a guarded A/B test.

Make LLM systems cheaper and faster: caching, routing, batching and GPU serving.

Part 6
Cutting LLM Cost by 70% Without Losing Quality
How to stack prompt caching, model routing, batch APIs and token hygiene into a 70-80% cost cut, with an eval gate that proves quality held.
Part 7
Reducing Time to First Token in a Chat App
Decompose TTFT into network, pre-model work, queueing and prefill, then cut each stage: parallelism, prefix caching, chunked prefill and headroom.
Part 8
Designing a Semantic Cache with Hit-Rate Tracking
A layered semantic cache for LLM responses: scoped keys, tuned thresholds, borderline verification, invalidation, and false-hit-aware hit-rate metrics.
Part 9
Routing Requests Between a Cheap and a Strong Model
Design an LLM router that sends each request to the cheapest good-enough model: rules, learned routers, cascades, threshold tuning and drift monitoring.
Part 21
Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization
How to size and run an open-weight LLM fleet: prefill vs decode, KV-cache math, continuous batching, FP8/FP4 quantization and goodput per dollar.

Reliable tool calling, agent loops, long-term memory and structured output.

Part 10
Building Tool-Using Agents That Work Reliably
Tool-use reliability is a systems property: tool design, a guarded control loop, permission gates, and pass^k evals on verifiable end states.
Part 11
Detecting and Recovering from Agent Loops
Classify why an agent loops, add harness guardrails and better tool feedback, and recover with an escalation ladder you can measure.
Part 12
Designing Long-Term Memory for an LLM Assistant
A tiered memory design with explicit write, read and forgetting policies, plus the benchmarks and production signals that show it works.
Part 13
Guaranteeing Valid JSON Output from LLMs
Enforce JSON Schema at decode time, design schemas that keep quality high, and add validation, repair and truncation handling on top.

Eval suites, LLM-as-a-judge and keeping hallucinations in check.

Part 14
Detecting and Reducing LLM Hallucinations
Separate hallucination types, detect them with claim-level verification and uncertainty signals, and reduce them without making the model useless.
Part 15
Building an Eval Suite for a Customer Support Bot
Build support-bot evals from real failures: layered offline tests, calibrated judges, statistically sound release gates and a production flywheel.
Part 16
Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift
Treat an LLM judge like a measurement instrument: calibrate it against human labels, remove its known biases, and size your eval sets for real error bars.

Defend agents against prompt injection and keep personal data away from models.

Part 17
Defending Tool-Using Agents Against Prompt Injection by Design
Why model-level filters can't stop prompt injection, and how control/data separation, taint tracking and sink policies make attacks harmless.
Part 18
Building a PII Redaction Gateway in Front of Your LLM
A layered design for detecting, pseudonymizing and re-identifying PII before it reaches a model, and how to measure the leak rate honestly.

When to fine-tune instead of prompting or RAG, and how to do it with LoRA.

Part 19
Fine-Tuning vs RAG vs Prompting: An Eval-First Decision Procedure
Diagnose whether your gap is knowledge or behavior, then climb from prompting to RAG to fine-tuning only when the cheaper rung plateaus.
Part 20
Fine-Tuning a Small Model with LoRA on 5,000 Examples
A practical LoRA recipe for an 8B model on 5K examples: data hygiene, hyperparameters, a real GPU memory budget, and eval-driven checkpoint selection.

Streaming, tracing, safe rollbacks and running chat for millions of users.

Part 22
Streaming LLM Responses over SSE and Surviving Disconnects
Stream tokens with server-sent events, decouple generation from the connection, resume with Last-Event-ID, and cancel cleanly so GPUs are freed.
Part 23
Tracing Every Step of a Multi-Agent Workflow
Model multi-agent runs as one OpenTelemetry trace: GenAI span conventions, context propagation across queues, private content storage and tail sampling.
Part 24
Rolling Back a Bad Prompt Change in Production
Make prompt rollback a sub-minute alias flip, catch regressions with canaries and guardrail metrics, and clean up what the bad prompt left behind.
Part 25
Designing a ChatGPT-Style Chat Service for 1M Daily Users
Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.