Tagged: GPU

Articles

• Part 7Cost, Latency & Serving

  Reducing Time to First Token in a Chat App

  Decompose TTFT into network, pre-model work, queueing and prefill, then cut each stage: parallelism, prefix caching, chunked prefill and headroom.

• Part 20Fine-tuning & Adaptation

  Fine-Tuning a Small Model with LoRA on 5,000 Examples

  A practical LoRA recipe for an 8B model on 5K examples: data hygiene, hyperparameters, a real GPU memory budget, and eval-driven checkpoint selection.

• Part 21Cost, Latency & Serving

  Serving Open-Weight LLMs at Scale: Batching, KV Cache, Quantization

  How to size and run an open-weight LLM fleet: prefill vs decode, KV-cache math, continuous batching, FP8/FP4 quantization and goodput per dollar.