Tagged: Streaming

Articles

• Part 7Cost, Latency & Serving

  Reducing Time to First Token in a Chat App

  Decompose TTFT into network, pre-model work, queueing and prefill, then cut each stage: parallelism, prefix caching, chunked prefill and headroom.

• Part 22Production Engineering

  Streaming LLM Responses over SSE and Surviving Disconnects

  Stream tokens with server-sent events, decouple generation from the connection, resume with Last-Event-ID, and cancel cleanly so GPUs are freed.

• Part 25Production Engineering

  Designing a ChatGPT-Style Chat Service for 1M Daily Users

  Back-of-envelope sizing, architecture and overload strategy for an LLM chat product at 1M DAU, where GPUs set both the cost and the latency.