Tagged: Latency

Articles

• Part 5Retrieval & RAG

  Adding a Reranker and Proving It Helped

  How to add a cross-encoder reranker to a RAG pipeline, size it, and prove the gain with paired offline evals, end-to-end checks and a guarded A/B test.

• Part 7Cost, Latency & Serving

  Reducing Time to First Token in a Chat App

  Decompose TTFT into network, pre-model work, queueing and prefill, then cut each stage: parallelism, prefix caching, chunked prefill and headroom.

• Part 21Cost, Latency & Serving

  Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization

  How to size and run an open-weight LLM fleet: prefill vs decode, KV-cache math, continuous batching, FP8/FP4 quantization and goodput per dollar.

• Part 22Production Engineering

  Streaming LLM Responses over SSE and Surviving Disconnects

  Stream tokens with server-sent events, decouple generation from the connection, resume with Last-Event-ID, and cancel cleanly so GPUs are freed.