Tagged: LLM-as-judge

Articles

• Part 2Retrieval & RAG

  Debugging a RAG System That Gives Wrong Answers

  A stage-by-stage method for localizing RAG failures: build a replayable failure set, attribute each case to a stage, fix the biggest bucket, and gate on it.

• Part 9Cost, Latency & Serving

  Routing Requests Between a Cheap and a Strong Model

  Design an LLM router that sends each request to the cheapest good-enough model: rules, learned routers, cascades, threshold tuning and drift monitoring.

• Part 14Evaluation & Reliability

  Detecting and Reducing LLM Hallucinations

  Separate hallucination types, detect them with claim-level verification and uncertainty signals, and reduce them without making the model useless.

• Part 15Evaluation & Reliability

  Building an Eval Suite for a Customer Support Bot

  Build support-bot evals from real failures: layered offline tests, calibrated judges, statistically sound release gates and a production flywheel.

• Part 16Evaluation & Reliability

  Making LLM-as-a-Judge Trustworthy: Calibration, Bias and Drift

  Treat an LLM judge like a measurement instrument: calibrate it against human labels, remove its known biases, and size your eval sets for real error bars.