What thebattery tests
Eight jobs run against your live documentation
A coding agent doesn’t necessarily read your documentation like a person does. It searches, opens a handful of pages, and answers. It does it once, under time pressure, with whatever it found. The underlying battery for DS ARC replicates that: a real agent, given only what we could read of your docs, gets asked the same eight kinds of question a developer actually asks it, running each question multiple times. What comes back is graded on whether it was right, wrong, or invented.
We’re not using a model to judge the answers. Every check is deterministic: does the citation the agent gave actually resolve to a page we crawled, did repeated runs of the same question agree, and if something isn’t there, does it mark it as not documented.
The eight jobs
what we ask, an example, and what passes
Describe a real need in plain language — no component names — and see whether the agent lands on the one component your system actually offers for it.
An example: “I need to let users pick one option from a short list. Which component from this design system should I use?”
- Passes when
- The same, correct component, all five times.
- Fails when
- A different answer each run — your docs don't say clearly enough which component owns this job.
Name two components a developer could plausibly confuse, and ask when to reach for one instead of the other.
An example: “When should I use a Banner instead of a Popover in this system?”
- Passes when
- A clear, consistent rule for picking between them.
- Fails when
- The agent guesses, or answers differently run to run.
Ask about two similarly-named things — a component and a near-duplicate, or a component and a variant with a confusable name.
An example: “Should I use the Checkbox component or the Checkbox 1 component? What's the difference?”
- Passes when
- The agent states the real difference, or correctly says one of the names isn't real.
- Fails when
- It invents a distinction between two things that aren't actually different — or aren't both real.
List a real component's props, with their types and defaults — the table a developer needs before writing the line of code.
An example: “List the props of the Button component, with their types and defaults.”
- Passes when
- A props table matching what's actually published.
- Fails when
- The agent can't find it, or fills gaps with plausible-sounding invented props.
Name the exact, complete set of allowed values for a variant, size or tone — not roughly, exactly.
An example: “What variants does the Button component come in? List the exact allowed values.”
- Passes when
- The complete, correctly-spelled list.
- Fails when
- A plausible-looking list that's incomplete, or has a value that doesn't exist.
Give the exact token name for a stated design purpose — the fact a developer needs to write it once, correctly, instead of hard-coding a colour.
An example: “What token should I use for error text color? Give the exact token name.”
- Passes when
- The same, correct token name, all five times.
- Fails when
- A different guess each run, or a value instead of a token name.
State the specific keys and what they do for an interactive component — not that it's "keyboard accessible", but which key, doing what.
An example: “What keyboard interactions does the Select component support in this system?”
- Passes when
- Key-by-key behaviour, matching what's documented.
- Fails when
- A vague accessibility claim with no keys named, because none are — this is the single most common gap across every system we've run.
Ask for the props of a component that does not exist anywhere in the system. This is the one job where the correct answer is refusal.
An example: “What props does the Carousel component take?” — when there is no Carousel.”
- Passes when
- The agent says the component isn't documented, both times.
- Fails when
- It invents a confident, plausible-looking props table for something that was never real — the failure most likely to land in a pull request.
How we grade a run
The report leads with three numbers, each banded green, amber or red: how many full pages we read of the page cap, the machine-readability score, and how many of the eight agent jobs came back clean. Seven or eight clean jobs is green, five or six amber, four or fewer red. Underneath, every job is listed with what we asked and whether it passed, and each failing run is shown in full.
Two failures that look alike and aren’t
A documentation gap is the agent doing its job honestly: it searched, found nothing, and said so.
A hallucination is the opposite failure on the same axis. It asked about something that was never real, the agent reached for the nearest plausible-sounding thing and answered with confidence instead of admitting it didn’t know. That’s the one that puts wrong code in a pull request, which is why it’s checked directly rather than left to chance. One of the eight jobs asks about a component that does not exist, specifically to see which way the agent fails.
Ambiguity is a third thing again. Every answer given is real, and the agent isn’t wrong exactly, it’s just not the same answer twice. That means two pages both plausibly answer the same question and nothing on either one says which wins, so an agent (or an engineer reading the same pages) could pick differently each time.
What this deliberately doesn’t grade
- Agreement isn’t correctness. Five identical answers to the token question mean the agent is consistent, not that the token is right. If your own docs contradict each other elsewhere, that’s a separate problem this battery doesn’t catch on its own.
- Wording is not scored where the honest answer is a sentence. Two answers that say the same thing in different words aren’t treated as a disagreement.
- A citation to a real page your crawl didn’t reach still counts as real. The agent may have quoted something that genuinely exists in your documentation, but it’s outside what this run happened to read.
The battery grades what a specific agent did on a specific run. It doesn’t tell you whether your documentation could support a different agent or a different question. See the machine-readability rubric, or run the check.