An Agent End to End: A Docs-Maintenance Agent, Measured

TL;DR

This finale builds one agent that does a real job: it reviews a documentation page for broken cross-references, code examples that no longer run, and sentences that contradict the library’s source code, then proposes a fix for each. It is built from the parts of this series: strict tools, adaptive thinking at a set effort level, a verifier, small tool results, checkpoints and budgets, a sandbox with a permission policy, and evaluation with repeats. On 18 pages of the python-semver docs with 24 seeded defects, Claude Haiku 5.5 found 98.3% of them across 90 runs, every recorded fix passed its check, and it cost $0.0014 per page. It also found four real defects already in the shipped docs, one of which a hand audit had missed. It caught only 60% of a harder set of facts that only the source code contradicts, and it tended to stop looking once the tools had handed it something to fix.

Why it matters

In 2013, a tool called DocRef checked the documentation of five widely used API libraries and found “more than 1,000 new documentation errors,” such as “broken code names and obsolete code samples.” Documentation rots because nothing runs it. The code changes, the tests change with it, and the page that explains the code keeps describing last year’s behavior.

That is a good job for an agent: tedious, in need of reading and judgment, and safe, because a human reviews the patch. It is also a good test of this series. Each part measured one piece of an agent: the loop, tool calls, thinking, verifiers, context, reliability, sandboxes and evaluation. This part puts them in one harness and asks two questions: does the result work, and which pieces earned their place?

The intuition

Think of a fact-checker at a magazine. Some checks are mechanical: does this phone number ring, does this quote appear in the transcript? A junior assistant with a script can do those. Other checks need reading: the article says the drug was approved in 2019, so someone has to find the approval letter and compare.

The docs agent works the same way. A link checker and a test runner do the mechanical checks, and what they report is always true. The model does the reading: it compares each sentence with the code that implements it, and it writes the fix.

The analogy breaks in two places. A fact-checker’s source is usually right; here the source of truth is code, which can itself have bugs, so “the page disagrees with the code” means “fix the page” only by policy. And a fact-checker who finds three errors on a page does not usually stop reading. This agent sometimes did.

Step 1: a docs folder with known defects

The corpus

The input is a copy of the documentation of python-semver, a small library for semantic version numbers (version 3.1.0, commit 15a42d0), made in a scratch folder. The repo itself is never edited. Its docs are written in reStructuredText (a plain-text markup format) and built with Sphinx, the standard Python documentation generator.

Three things on a page can go stale. A cross-reference is a link written as a role, such as :meth:`~semver.version.Version.bump_major` for a method or :ref:`sec.replace.parts` for a section label; it breaks when the target is renamed. A doctest example is a snippet in the style of an interactive Python session, a >>> line followed by the output it should print; Python’s doctest module “executes those sessions to verify that they work exactly as shown.” And a factual claim is a sentence such as “By default (False), the version string to parse has to follow the semver specification”, which only the code can confirm.

The agent reviews the 18 pages of the “Usage” and “Migration” sections: 3,273 words, 108 cross-references and 156 doctest examples. The source of truth is the library itself, src/semver/ (1,608 lines of Python) plus its changelog.

Seeding defects

To measure recall (the share of real defects found) and precision (the share of reports that are real), you need to know where the defects are. So a script seeds 24 of them, each a one-line edit, eight of each kind:

Half of the fact seeds are also contradicted by an example on the same page; the other half only by the source code. A second, harder set of six fact seeds goes into a separate copy: claims that only the source contradicts, with no example nearby. One says an incomplete dictionary’s missing parts become None when compared (they become zero), another that a match expression without an operator means “at least this version” (it means “equal”).

Scoring

A report matches a seed if it is on the same page within three lines of it. A fix counts as correct if, applied to the page, the broken link resolves or the failing example passes with nothing new broken, or, for facts, if the edited lines no longer contain the wrong claim and do contain the right one (checked with a pattern per seed). Everything else gets read by hand and labeled as either a real defect that was already in the docs or a false positive.

Before running anything, a hand audit of the clean pages found three real defects in the shipped docs: two cross-references to Version.isvalid, a method that no longer exists, and a page saying match() “expects two arguments: a version string and a match expression” when it takes one. The agent later found a fourth.

Step 2: the agent, built from the series

The agent reviews one page per run. Pages are independent, so there is no orchestrator; the runner runs three pages at a time. The lab module is finale.py, 364 lines on top of the Part 1 harness and Part 12’s durable loop.

The model never writes to the docs: its only output is a list of findings that becomes a patch.

Seven tools

read_doc and read_source return numbered line ranges; search_source greps the library. check_links resolves every cross-reference against the docs tree and the imported library and suggests close matches. run_examples runs the page’s doctests. check_fix is the verifier, and report_issue records a finding. Its schema forces a fix:

@tool("Report one defect on the page under review, with a proposed fix. kind is 'link', 'code' or "
      "'fact'. line is the line number where the defect is. old_text is copied exactly from the page "
      "and must occur once; new_text replaces it. Reporting the same page, line and kind again "
      "replaces the earlier report.",
      path={"type": "string"}, line={"type": "integer"},
      kind={"type": "string", "enum": ["link", "code", "fact"]},
      problem={"type": "string", "description": "One sentence: what is wrong and what the code says."},
      old_text={"type": "string"}, new_text={"type": "string"})

Why the checks run in a sandbox

check_fix runs whatever doctest the model writes into its fix. That is model-written code, so it runs inside bubblewrap, a small Linux sandboxing program and the same one Part 13 measured, with every namespace unshared, an empty environment and the corpus mounted read-only:

cmd = ["bwrap", "--unshare-all", "--die-with-parent", "--new-session", "--clearenv",
       "--setenv", "PATH", "/usr/bin:/bin", "--setenv", "PYTHONPATH", "/corpus/src",
       "--setenv", "PYTHONDONTWRITEBYTECODE", "1",
       "--ro-bind", "/usr", "/usr", "--symlink", "usr/bin", "/bin",
       "--symlink", "usr/lib", "/lib", "--symlink", "usr/lib64", "/lib64",
       "--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp",
       "--ro-bind", str(corpus), "/corpus", "--ro-bind", str(CHECKERS), "/checkers"]

A hand-written fix that tried to escape confirmed it: inside, the home directory did not exist, writing to the docs raised “Read-only file system,” and opening a socket raised “Network is unreachable.” Each check took a median of 47 ms (links) and 55 ms (examples) in the sandbox against 38 ms and 42 ms without it, over 30 runs each, so isolation added 10 to 14 ms to a step whose model request, including the ledger’s token count, took a median of 3.7 seconds.

The permission policy is short, because the agent can’t change anything: it denies any path outside the corpus, any read of a file that isn’t library source, and any check_fix or report_issue on a page other than the one under review.

Step 3: what it finds, misses and costs

Setup

Model claude-haiku-5-5, effort low, adaptive thinking, max_tokens 8,000. All 18 pages, 5 runs each: 90 runs. Every run made its API calls from this laptop over the public API, three pages at a time, on October 25, 2026; API latency varies with the time of day. Costs come from the lab ledger. Intervals are 95% Wilson confidence intervals; differences between configurations use a paired bootstrap over seeds, because the runs of one seed are not independent (Part 11 explains why).

Results

None of the 128 reports was wrong. Each was a seed or one of the two broken isvalid links that shipped in the real docs, and every fix was applied and checked, not just read.

Most of the bill was output. Prompt caching served 89.7% of the 2.57 million input tokens the runs read, so input cost about $0.056 of the $0.13 and the 147,239 output tokens, mostly thinking, cost $0.074.

The two misses were one tool problem

Both missed seeds were C2, an example whose expected output -1 had been changed to 1. The agent saw the failure every time; it could not record a fix. The page indents the example by 4 spaces, the model kept copying it with 6, and report_issue refused text that wasn’t on the page. One run ended with:

- **Not recorded (code, line 15):** `semver.compare("1.0.0", "2.0.0")` returns `-1`, but the page
  expects `1`. The fix is to change the expected output on line 15 from `1` to `-1`. Every attempt
  to match that two-line span failed with "occurs 0 times", so `report_issue` rejected it.

This was common: 49 of 170 check_fix calls (28.8%) failed on text that didn’t match the page, every one of them on whitespace. The verifier caught them before they became bad reports, which is its job, but the agent then had to retry or give up. A second version of the tools accepts a multi-line old_text whose lines differ from the page only in leading spaces, and keeps the page’s own indentation. Over another 90 runs it found C2 in 5 of 5, cut rejected check_fix calls to 22.8%, and found 118 of 120 seeds again: two fact seeds slipped instead.

What it misses: facts only the code contradicts

The seeded facts were too easy: half sat next to an example that contradicted them. On the hard set, six claims only the source code contradicts, the same configuration found 18 of 30 (60.0%, 42.3-75.4%). It never caught H1, the claim that missing dictionary parts become None; every run read the comparison code and said the page had no defects; one concluded that “the dict case behaves as described.” It caught the versionadded number (H6) and the dictionary keys rule (H2) in only 2 of 5 runs each.

The third real defect in the shipped docs, the page saying match() takes two arguments, shows the same weakness from another angle. On the seeded corpus that page also carries two seeds that check_links flags at once. The agent fixed those and reported the match() sentence in 0 of 5 runs. On the clean copy of the same page, with nothing for the tools to flag, it reported it in 2 of 3 runs. When the agent ran without the checker tools at all, it reported it in 4 of 5. The numbers are small, but they point one way: when the tools hand a low-effort agent something to fix, it fixes that and stops reading. Anthropic’s effort docs warn about exactly this for Haiku 5.5: at low, “the model is more likely to skip a search, stop early, or skip a check.”

Real defects on the clean pages

On the clean pages (3 runs of each of the 18), the agent made 11 reports. Two were false positives. One argued that VersionInfo isn’t really deprecated because no warning is raised, a reading of the code but not a defect in the page. The other was junk: a “finding” whose problem read “No defect found; placeholder not reported” with old_text equal to new_text, which a one-line harness check would reject.

The other nine were real, and one was new. The page on raising versions says that if “the last part of the dot-separated build metadata is not numeric, .0 is appended.” The agent pointed out that _increment_string increments the last run of digits anywhere in the string and appends .0 only when there is none. It is right: 3.4.5+1abc bumps to 3.4.5+2abc, with no .0. The pre-run audit for this article had missed it.

Step 4: what each piece was worth

The same pages ran under four more configurations and a no-model baseline: 612 agent runs in all, for $2.33. Seeded results are per seed per run; “real defects” counts how often each of the four defects already in the docs was reported, per run of its page, on the clean copy.

Haiku low ran 5 repeats on the seeded and hard pages and 3 on the clean ones; medium ran 3, 5 and 3; Sonnet 3, 5 and 2.

The checkers are most of the job

With no model at all, the two scripts flagged all 16 link and code seeds and both broken isvalid links. A simple rule fixed 12 of the seeds: use the closest existing target for a link, and replace an example’s expected output with what the code printed. The rule failed where judgment was needed. For the three examples that call a renamed method, it either pasted the AttributeError in as the new expected output or had nothing to offer; the agent changed the call back to the real method name, every time. And no script touches a single fact.

Taking the scripts away from the agent cost 11.7 points of recall (95% interval 3.3 to 21.7 points), all of it on links and examples: it found 77.5% of the broken links and 82.5% of the broken examples by reading. It missed sec.convert.version against sec.convert.versions in 5 of 5 runs, the kind of one-letter difference a human eye skips too. It also made six false reports, against none with the scripts, including one that pinned a correct link bug on the wrong line.

Effort and model buy the hard facts

On the hard set, medium effort found 80.0% (62.7 to 90.5%) against low’s 60.0%, a gain of 20 points (6.7 to 36.7) for 1.5 times the cost per page, $0.0019 against $0.0012. Sonnet 5.5 at low found 86.7% (70.3 to 94.7%), 26.7 points above Haiku low (6.7 to 46.7) at 13 times the cost, $0.0164 per page. Haiku medium and Sonnet are within noise of each other on these 30 trials. Both still missed H1, the dictionary claim, in 4 of 5 runs.

Neither fixed the distraction from Step 3. On the seeded match() page, where the tools flag two other defects, the real argument-count error went unreported in all 16 runs across every configuration with the checkers: Haiku low and lenient, medium, and Sonnet. On the clean copy of the page, medium reported it in 3 of 3 runs and Sonnet in 2 of 2.

Sonnet also behaved differently with the tools. It called check_fix 6 times in 54 runs, where Haiku called it 170 times in 90, and reported fixes without verifying them, against the prompt’s instructions. Its fixes were right anyway: 71 of 72 passed. A stronger model leaned less on the verifier, so the harness, not the prompt, has to decide whether an unverified fix is acceptable.

The safety and reliability pieces never fired

No run crashed, hit its 30-step or cost budget, or reported the same finding twice, so the checkpoints, budgets and idempotency key were never used. The most expensive Haiku page cost $0.0049 and the longest took 14 steps. The policy denied 1 of 4,404 tool calls, a read_source on docs/conf.py. On a job this short and this read-only, that is the expected result: these pieces are insurance, and Part 12 measured what happens without them when a run is long and its tools have side effects.

In the real world

The mechanical half of this job has had tools for years. Sphinx’s doctest builder runs marked code blocks as tests, and python-semver already runs its page examples under pytest, so a seeded code defect would fail its CI before release. Sphinx’s linkcheck builder checks only external links; for internal cross-references there is nitpicky mode, in which “Sphinx will warn about all references where the target cannot be found.” python-semver’s configuration doesn’t enable it, which is how two references to the removed isvalid method reached the published docs.

Facts are the part no build step checks. DocRef went after them in 2013 with natural-language parsing plus code analysis, and found more than 1,000 errors in five libraries. The agent here does the same comparison by reading, at about a tenth of a cent per page.

The design follows Anthropic’s advice in “Building effective agents”: “finding the simplest solution possible, and only increasing complexity when needed.” Two of the three checks are plain scripts, the per-page runs are a fixed workflow, and the model only does what scripts can’t. The same post recommends “extensive testing in sandboxed environments,” which is what Steps 3 and 4 did.

The series in one table

Each part of this series measured something. This is the one result from each that shaped the agent above.

Common misconceptions

• “A 98% score on seeded defects means the agent is 98% reliable.” It means the seeds were easy. Half the seeded facts sat next to an example that contradicted them. On six facts only the code contradicts, the same agent found 60%. A benchmark at its ceiling can’t tell two configurations apart; you need a set it fails.
• “The model should do the checking.” Without the link checker and the example runner, the agent found 77.5% of the broken links and 82.5% of the broken examples instead of 100% and 95%, and made six false reports instead of none. A script that resolves every reference doesn’t skim; give it the mechanical checks and spend the model on reading.
• “More tools means more coverage.” The tools changed where the agent looked. On the page where check_links flagged two seeds, the low-effort agent fixed them and reported the real match() error in 0 of 5 runs; without the checkers it reported it in 4 of 5.
• “A run that reports nothing has checked everything.” On the clean copy of the docs, the agent missed the real build-metadata error in 2 of 3 runs of that page and said the page had no defects. A quiet run is a claim like any other: sample those runs and check them too.
• “A stronger model makes the harness less important.” Sonnet 5.5 found more of the hard facts, but it also skipped the verifier: 6 check_fix calls in 54 runs, against the prompt’s instructions. Its fixes happened to be right. The guarantees that hold whatever the model does, such as no write access, a sandbox and a check before a patch is applied, live in the harness.

Key takeaways

• A useful agent is mostly harness. Here, scripts did the link and example checks, a sandbox ran the model’s code, and the model did the one thing scripts can’t: read a sentence against the code and write the fix.
• Built that way, Claude Haiku 5.5 found 98.3% of 120 seeded defects with no false reports, for about a tenth of a cent per page, and found a real error in the shipped docs that a human audit missed.
• Seeded defects that are too easy tell you nothing. On facts only the code contradicts, the same agent found 60%; medium effort raised that to 80% for 1.5 times the cost, close to Sonnet 5.5’s 87% at 13 times the cost.
• Watch where the agent stops. When the tools handed it defects on a page, it fixed those and missed the real error in the prose, in all 16 runs with the checkers turned on.
• Measure the failures you can fix in the harness. Text matching, not reasoning, caused both misses in the main run, and a no-op report and an unverified fix are each one line of harness code away from being rejected.

What’s next

This is the last part of Agentic AI. If you want to know what happens inside the model that made these tool calls, start with LLM Fundamentals; for serving it fast and cheaply, LLM Inference; for training one yourself, Build an LLM From Scratch; and for building systems around it, System Design: Building Tool-Using Agents That Work Reliably. A later series, Measuring LLMs, will cover how evals and benchmarks are built, and how far to trust them.

Further reading

• Detecting API Documentation Errors (Zhong and Su, OOPSLA 2013): DocRef, which combined language and code analysis and found more than 1,000 errors in five libraries’ docs.
• doctest (Python docs): runs interactive examples in text to check they still work as shown.
• sphinx.ext.doctest, Builders and nitpicky: the build-time checks for code blocks, external links and internal references.
• Building effective agents (Anthropic, 2024): workflows versus agents, and starting from the simplest design.
• Effort (Claude API docs): effort levels on Claude Haiku 5.5 and Sonnet 5.5, and what low gives up.
• Pricing (Claude API docs): the per-token prices behind every cost in this article.
• python-semver: the library whose docs this agent reviewed.
• Building Tool-Using Agents That Work Reliably: the System Design view of the same harness.