Production Engineering

Articles

• Part 22

  Streaming LLM Responses over SSE and Surviving Disconnects

  Stream tokens with server-sent events, decouple generation from the connection, resume with Last-Event-ID, and cancel cleanly so GPUs are freed.

• Part 23

  Tracing Every Step of a Multi-Agent Workflow

  Model multi-agent runs as one OpenTelemetry trace: GenAI span conventions, context propagation across queues, private content storage and tail sampling.

• Part 24

  Rolling Back a Bad Prompt Change in Production

  Make prompt rollback a sub-minute alias flip, catch regressions with canaries and guardrail metrics, and clean up what the bad prompt left behind.

• Part 25

  Designing a ChatGPT-Style Chat Service for 1M Daily Users

  Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.