Tagged: Observability

Articles

• Part 2Retrieval & RAG

  Debugging a RAG System That Gives Wrong Answers

  A stage-by-stage method for localizing RAG failures: build a replayable failure set, attribute each case to a stage, fix the biggest bucket, and gate on it.

• Part 8Cost, Latency & Serving

  Designing a Semantic Cache with Hit-Rate Tracking

  A layered semantic cache for LLM responses: scoped keys, tuned thresholds, borderline verification, invalidation, and false-hit-aware hit-rate metrics.

• Part 11Agents & Tool Use

  Detecting and Recovering from Agent Loops

  Classify why an agent loops, add harness guardrails and better tool feedback, and recover with an escalation ladder you can measure.

• Part 15Evaluation & Reliability

  Building an Eval Suite for a Customer Support Bot

  Build support-bot evals from real failures: layered offline tests, calibrated judges, statistically sound release gates and a production flywheel.

• Part 16Evaluation & Reliability

  Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift

  Treat an LLM judge like a measurement instrument: calibrate it against human labels, remove its known biases, and size your eval sets for real error bars.

• Part 18Security & Privacy

  Building a PII Redaction Gateway in Front of Your LLM

  A layered design for detecting, pseudonymizing and re-identifying PII before it reaches a model, and how to measure the leak rate honestly.

• Part 23Production Engineering

  Tracing Every Step of a Multi-Agent Workflow

  Model multi-agent runs as one OpenTelemetry trace: GenAI span conventions, context propagation across queues, private content storage and tail sampling.

• Part 24Production Engineering

  Rolling Back a Bad Prompt Change in Production

  Make prompt rollback a sub-minute alias flip, catch regressions with canaries and guardrail metrics, and clean up what the bad prompt left behind.