Tagged: Architecture

Articles

• Part 1Retrieval & RAG

  Designing a RAG System for 10 Million Documents

  How to size, build and measure retrieval-augmented generation over 10M documents: ingestion, hybrid retrieval, reranking, ACLs, cost and evals.

• Part 17Security & Privacy

  Defending Tool-Using Agents Against Prompt Injection by Design

  Why model-level filters can't stop prompt injection, and how control/data separation, taint tracking and sink policies make attacks harmless.

• Part 18Security & Privacy

  Building a PII Redaction Gateway in Front of Your LLM

  A layered design for detecting, pseudonymizing and re-identifying PII before it reaches a model, and how to measure the leak rate honestly.

• Part 19Fine-tuning & Adaptation

  Fine-Tuning vs RAG vs Prompting: An Eval-First Decision Procedure

  Diagnose whether your gap is knowledge or behavior, then climb from prompting to RAG to fine-tuning only when the cheaper rung plateaus.

• Part 22Production Engineering

  Streaming LLM Responses over SSE and Surviving Disconnects

  Stream tokens with server-sent events, decouple generation from the connection, resume with Last-Event-ID, and cancel cleanly so GPUs are freed.

• Part 24Production Engineering

  Rolling Back a Bad Prompt Change in Production

  Make prompt rollback a sub-minute alias flip, catch regressions with canaries and guardrail metrics, and clean up what the bad prompt left behind.

• Part 25Production Engineering

  Designing a ChatGPT-Style Chat Service for 1M Daily Users

  Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.