From 31 wrong or borderline checks in every 100 to 4: how we tuned Blue Pencil for our own technical notes
This is how we at Spectacular AI use Blue Pencil internally: our AI agents write the notes that document our own code, and Blue Pencil measures every draft before anyone reads it.
At a glance
An AI agent tuned our checks until we could trust them. Then AI agents used them to fix new drafts.
314
Checks wrong or borderline: 31 of every 100 at the start, and 4 of every 100 after tuning.
202–3
On one note: a review of one note runs 63 checks, so at those rates about 20 would be wrong or borderline at the start, and 2 or 3 at the end.
37100
Rules met by a new draft: 37 of every 100 on a first draft written from a one-line request.
After one round of fixes: all 63 rules met, on each of three new notes.
$90.70$11.74
Cost of a round of tuning: $90.70 for the first round, and $11.74 for the third.
The challenge
Our first rules for checking our own notes were wrong or borderline on 31 of every 100 checks.
The notes are part of a wiki about our own product. Each data-schema note describes one kind of record our code writes or reads, with every field, its type and the rules its values follow. People and AI coding agents read these notes before they change the code.
We wanted AI agents to write new notes, and we wanted every draft checked against one written definition of a good note before anyone read it. The first rules looked reasonable until we ran them on four notes we knew were right.
What we did
We tuned the rules in three rounds, and an AI agent did the tuning. It followed only the written guide that comes with Blue Pencil. After each round we improved the guide, and a new agent started from scratch.
Each round repeats three steps:
Write the rules for what a good note has.
Check the rules on the four notes we knew were right, and on copies of those notes with one planted mistake each.
Rewrite the rules that came back wrong or borderline, and check again.
The planted mistakes told us which rule should fail on each copy, so every check had a known right answer.
We got stuck at 88 of every 100 checks right and clear, from the end of round 1 to the start of round 3. The last rules were too vague to give the same answer twice. Once the agent rewrote them precisely, we reached 96.
Results
The checks became reliable. Over three rounds and eleven passes, the checks that were right and clear went from 69 of every 100 to 96 of every 100.
New drafts met the rules after one round of fixes. We gave an agent three short fact sheets for new notes. Asked in one line to write each note, its first drafts met 37 of every 100 rules: each one left out 5 to 7 whole sections. After one round of automatic fixes, guided by Blue Pencil's checks, all three drafts met every rule.
A detailed brief helped, and the checks still caught a gap. With a detailed written brief instead, the first drafts met 98 of every 100 rules. The one rule each draft missed was one the brief never stated.
One note, before and after. For this page we ran the test again on a new note, about the request our review route takes. We asked an AI model for it in one line, and its first draft met 19 of its 63 checks: it left out 5 of the parts a good note has. The model fixed it once, guided only by Blue Pencil's checks, and the note met 61 of 63.
- Overview
- Definition
- Fields
- Producers and Consumers
- Rules
- Response
- Frontmatterleft out
- Writers and readers
- Fields
- Relationsleft out
- Constraints
- Exampleleft out
- Defined inleft out
- Change logleft out
"We use Blue Pencil internally on the notes that document our own code. Our first rules looked fine, and 31 of every 100 checks came back wrong or borderline. Three rounds later the checks were right and clear on 96 of every 100, and an agent's drafts from a one-line request met every rule after one round of fixes."
What it took
Tuning took three rounds. The first round cost $90.70 of AI usage and the third cost $11.74, because we improved Blue Pencil's tools and its guide between rounds.
Some rules still get a borderline answer, such as a rule about every row of a table or the exact form of a link. Blue Pencil's guide lists those rules, so you can leave them out or check them yourself.
Our new note for this page did not meet every rule. After one round of fixes, 2 checks still failed: its example did not say where it came from, and one relation did not name another record. We ran a second round, which fixed both, but the check that finds the frontmatter then came back just below the line, so its 4 checks were not asked: 58 of 63. The second round did worse than the first.
These results cover one kind of document, three new notes and the one above. An AI agent fixed the drafts, and the same checker that guided the fixes scored them. So the results show that the drafts meet our definition of a good note, not that people prefer them.