Part 1Retrieval & RAG
Designing a RAG System for 10 Million Documents
How to size, build and measure retrieval-augmented generation over 10M documents: ingestion, hybrid retrieval, reranking, ACLs, cost and evals.
New series · LLM System Design
Build LLM systems that hold up in production
25 in-depth articles on retrieval, evaluation, agents, serving, security and operations. Each one gives you the design, the numbers that size it, the ways it fails, and the sources to check.
Start here
Three articles that cover the ground most LLM products stand on.
Part 6Cost, Latency & Serving
Cutting LLM Cost by 70% Without Losing Quality
How to stack prompt caching, model routing, batch APIs and token hygiene into a 70-80% cost cut, with an eval gate that proves quality held.
Part 25Production Engineering
Designing a ChatGPT-Style Chat Service for 1M Daily Users
Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.
Explore by topic
Seven areas you need to get right before an LLM feature is ready for real traffic.
Retrieval & RAG
Chunking, hybrid search, reranking and debugging retrieval-augmented generation.
Cost, Latency & Serving
Make LLM systems cheaper and faster: caching, routing, batching and GPU serving.
Agents & Tool Use
Reliable tool calling, agent loops, long-term memory and structured output.
Evaluation & Reliability
Eval suites, LLM-as-a-judge and keeping hallucinations in check.
Security & Privacy
Defend agents against prompt injection and keep personal data away from models.
Fine-tuning & Adaptation
When to fine-tune instead of prompting or RAG, and how to do it with LoRA.
Production Engineering
Streaming, tracing, safe rollbacks and running chat for millions of users.
The full reading order
Twenty-five deep dives on building production LLM systems, from retrieval and evaluation to serving, security and operations.
- 01Designing a RAG System for 10 Million DocumentsRetrieval & RAG
- 02Debugging a RAG System That Gives Wrong AnswersRetrieval & RAG
- 03Chunking Legal Contracts for RetrievalRetrieval & RAG
- 04BM25 vs Dense vs Hybrid Search: When to Use EachRetrieval & RAG
- 05Adding a Reranker and Proving It HelpedRetrieval & RAG
- 06Cutting LLM Cost by 70% Without Losing QualityCost, Latency & Serving
- 07Reducing Time to First Token in a Chat AppCost, Latency & Serving
- 08Designing a Semantic Cache with Hit-Rate TrackingCost, Latency & Serving
- 09Routing Requests Between a Cheap and a Strong ModelCost, Latency & Serving
- 10Building Tool-Using Agents That Work ReliablyAgents & Tool Use
- 11Detecting and Recovering from Agent LoopsAgents & Tool Use
- 12Designing Long-Term Memory for an LLM AssistantAgents & Tool Use
- 13Guaranteeing Valid JSON Output from LLMsAgents & Tool Use
- 14Detecting and Reducing LLM HallucinationsEvaluation & Reliability
- 15Building an Eval Suite for a Customer Support BotEvaluation & Reliability
- 16Making LLM-as-a-Judge Trustworthy: Calibration, Bias and DriftEvaluation & Reliability
- 17Defending Tool-Using Agents Against Prompt Injection by DesignSecurity & Privacy
- 18Building a PII Redaction Gateway in Front of Your LLMSecurity & Privacy
- 19Fine-Tuning vs RAG vs Prompting: An Eval-First Decision ProcedureFine-tuning & Adaptation
- 20Fine-Tuning a Small Model with LoRA on 5,000 ExamplesFine-tuning & Adaptation
- 21Serving Open-Weight LLMs at Scale: Batching, KV Cache, QuantizationCost, Latency & Serving
- 22Streaming LLM Responses over SSE and Surviving DisconnectsProduction Engineering
- 23Tracing Every Step of a Multi-Agent WorkflowProduction Engineering
- 24Rolling Back a Bad Prompt Change in ProductionProduction Engineering
- 25Designing a ChatGPT-Style Chat Service for 1M Daily UsersProduction Engineering