How we AI-updated 1,138 Error-code Docs Pages

How we AI-updated 1,138 Error-code Docs Pages

Here's a story I've been wanting to tell: we recently used AI to fix more than a thousand Twilio API error-code pages on our docs pages in about 12 weeks – and a human reviewed every single change. In this post, I’ll explain how we did it, what broke along the way, and the prompt we open-sourced so you can try the same thing on your own docs.

The short version: Score every page with a plain script first, so you only send genuinely bad ones to a model. Generate fixes with a prompt that's allowed to say "nothing to do here." Hash-check your code samples so the model can't quietly rewrite them. And – most importantly – keep a human on every diff. Want an example? Our reusable core is open-sourced.

Why did we revise our error codes, and why listen to us?

Error-code pages are the docs you read when something is already going wrong. That's why they matter: you need them at your worst moment. And that is exactly why they can't be allowed to rot: as products evolve and new causes emerge, out-of-date pages fail you right when you need them most. Ours were a mix of qualities, and inconsistent – it was time to dedicate our time to cleaning them up.

Twilio's error-code catalog is not a small, fixed thing to maintain: we power billions of interactions across an expanding ecosystem of global channels. We had roughly 2,692 error codes when we ran our audit, and we have more than 3,200 as this post went to press since new codes ship right along with new products. At our scale and pace, consistency has to be systematic. A writer polishing pages one at a time can't keep up with a catalog that changes every launch.

Before writing a line of tooling, we studied two awesome teams who already nail error docs:

Plaid ( plaid.com/docs/errors) has an amazing information architecture: one hub covers most-common errors, all errors, and the response schema, with every page on the same template (sample message, common causes, a troubleshooting checklist). They also have a nice "Was it helpful?" widget that helps tell them where a page is falling short.

Stripe ( docs.stripe.com/error-codes) won us over on layering: a basic SDK guide for beginners, a separate HTTP-level guide for folks working against the raw API, language-aware code samples, a clean taxonomy (api_error, card_error, invalid_request_error), and a separate decline codes reference for card-specific failures.

Those two helped shape the standards we held ourselves to in this effort: a consistent response format, a central registry at twilio.com/docs/api/errors, and one principle: one error code, one customer fix.

During our earlier messaging work, updating the documentation alone drove a 45% decrease in error volume. Alongside other changes, the error rate for those codes fell from 23% to 8.8%. Clearer error docs don't just read better, they measurably reduce the errors people hit in the first place.

Why docs-led, not eng-led

Why did our docs team take this on instead of each product's engineers? Two reasons: speed of coverage, and consistency.

Done by hand, we find improving an error page takes about two days of a writer's time: researching the error, reproducing it, writing causes and solutions, finding the links, and doing the writer-side review. That estimate doesn't include asking for or waiting on external reviews.

Our pipeline completed the generation pass across the catalog in about 11 hours – around 34.6 seconds per page. Across the whole catalog, that adds up: the manual path would take something like 18,000 hours, the better part of a decade for one person.

But speed isn't the real prize. Consistency is. With the docs team owning the pass, every page came out with the same structure, voice, and standards.

Architecture

Phase 1: a script that judges docs before touching them

The tempting first move with a pile of stale docs is to hand every page to an LLM and say "make these better." Please don't!

Most of the cost in editing – and most of the risk – comes from touching pages that are already fine. It’s better to score every page first and only send the ones that need work into the pipeline.

Our scorer is a plain script – so no LLM, no randomness. It read each page, and handed back a score out of 100 across three weighted dimensions:

• Structure (up to 50 points): are the required frontmatter fields present and valid, and are the expected sections there?
• Content (up to 35 points): is this real prose, or just a title and a placeholder? We set a floor: anything under 40 characters of body text doesn't count as content.
• Resources (up to 15 points): are there useful, correctly formatted links?

Every page started at full marks (100) and lost points for specific defects. Then we routed:

• below 70: went to the pipeline
• 70 to 84: got a human look
• 85-plus: we left it alone.

How did our error codes look?

We audited roughly 2,692 error-code pages and gave each one a score. That audit flagged 2,479 pages for update and pulled 213 out for manual review.

Now, we didn't try to fix them all at once. We sorted worst-first and worked in disciplined batches – at this point, we’ve completed 1,138.

Our exact rubric is proprietary, and (honestly) still hardcoded rather than config-driven. The prompt itself, though, is open source, part of a broader collection of docs-related prompts we've published at docs-ai-buddy.The four things you'd need to pull out in the config are weights, penalties, field manifest, and a link allowlist.

Phase 2: the LLM-driven pipeline

Once a page is flagged, it runs through a pipeline where every stage is boring on purpose:

Each stage is small and inspectable: Select picks the next candidate code, Generate calls the model (we used OpenAI's gpt-5.4 via their Responses API, using a prompt we keep versioned in OpenAI's dashboard rather than embedded in the code), Sanitize cleans the output, Patch applies it with git apply, Lint checks the result, and the last stage commits and opens a PR.

And git apply takes a diff, not a full file. The Patch stage builds one for you as a single hunk that deletes every old line and adds every new one.

We ran the batches on gpt-5.4, with a --model flag to drop to the smaller variants (gpt-5.4-mini, gpt-5.4-nano) on simpler pages. We treated a page as simple when the audit and source file suggested a low-risk rewrite: for example, it had sparse existing guidance, few links to preserve, and no code examples or complicated product-specific caveats. The calls ran through a small worker pool at a default concurrency of 4, with a 180-second per-call timeout (to prevent CLI timeouts).

The prompt itself is open-sourced, so let me invite you again: go take it.

Here are a few of the rules which have really held up so far on our updates:

1. Write in a direct, instructional voice.
2. Never fabricate claims or sources.
3. Cap additional resources at three links, and never link one error page to another.
4. And the most important: the model is allowed to do nothing. If a page can't be improved, it responds with exactly "I can't find anything to add to this page."

Human in the loop, across every phase

Humans didn't just sign off at the end. Our team gated every phase, and we built the tooling to protect our work – our named reviewers, Ana Benites, Ryan Chinn, and Elmer Thomas, owned technical validation. We started with small batches of around 30 pages per PR and scaled up to about 100 once we trusted the pipeline, keeping each batch small enough for reviewers to manage properly.

Two mechanics kept the automation from stepping on people:

• A tracking.json file records every completed code, and the pipeline skips anything listed there.
• A wrapper script also adds codes from in-flight PRs to that same file, so we never open a duplicate PR over work already in progress. The pipeline also skips any file with local modifications.

One thing that I really want to stress is important: there's no automated eval harness and no retrieval system behind any of this. The human review was our eval.

Deploying it to production

Of course, there were some growing pains with our process. Here are a few of the problems we hit – hopefully they can be useful if you work on a similar cleanup!

Issues you might hit

The model narrated its own reasoning into your docs. This was our single most common problem. The model would write hedging meta-commentary straight into customer-facing text, like "This is an inference based on Twilio only accepting specific TwiML verbs."

Human reviewers caught that pattern across a dozen pages, but even one page would have been enough. Readers don't want to watch a model think, they want the facts! For us, the fix was a mechanical rewrite: turn "This is an inference based on X" into "because X".

The rewrite would sometimes change what the error actually meant. (This was the scary one.) On error 63029, the original page said the receiver failed to download the template, which is a handset and delivery problem. The pipeline rewrite described a ContentVariables parameter mismatch, though, which is a sender-side problem.

Now, both explanations are plausible and well-written. This sort of error is where human-in-the-loop really matters: only a reviewer with real domain context and expertise can tell when meaning has drifted.

Code samples sometimes broke from two directions at once: in the model, and in our own pipeline. In developer docs this is a dangerous failure, because a code sample that's subtly wrong still looks completely trustworthy. The obvious risk is the model changing a code snippet while it's "improving" the words around it… and the sneakier risk was us.

After the model finished, one post-processing script ran a few tidy-up passes over the page, and two cleanup steps inside that script could reach into the code by mistake. One was a find-and-replace that made the writing less repetitive by swapping a phrase like "Error code 30007" for "this error." It was only meant to touch prose, but it edited matching text inside code samples too. The other trimmed leftover junk off the end of a page and could accidentally cut off a snippet that lived down there.

Either one could break a working example even when the model's output was otherwise perfect. To fix this class of errors, we added a safety check that took a fingerprint of every code snippet before the update and compared it afterward – if a snippet changed at all, the whole page was rejected. A rejected page never gets committed or opened as a PR, so no human ever reviews broken code as if it were fine. It just falls back into the queue, unchanged, to be picked up again on a later run.

Small stuff that only shows up at scale. We also hit a number of formatting issues and grammatical mistakes – stuff like run-on bullets, URL typos like a doubled Why-WWas-My-Toll-Free-Verification-Rejected, lowercase sentences after a period, and the like. We also saw style-guide drift where the pipeline used "check" where "verify" reads better, or the model output more than three resource links.

Any one of those mistakes was trivial, but across a thousand pages, they're a flood. We lint for each pattern before a human opens the PR, push as much of the style guide into the prompt as we can, and let a reviewer mop up the rest.

Results, honestly

Across roughly 27 pull requests between March and May 2026, about 12 weeks, we remediated 1,138 error codes, the count of codes marked complete in our tracking across those PRs.

Getting more than a thousand pages to a consistent, reviewed standard in a quarter was an incredible result for a catalog that would have taken the better part of a decade to review manually.

We're also not leaving the catalog to drift back to where it started. PMs and stakeholders shipping new error codes now align their PRs to the same error code standards from day one, so new codes don't get to start out the way the old ones did. And a teammate is already building on top of our experience: they’re making an automated error code generator that will draft documentation for brand-new codes the moment they're created, instead of letting them join next year's backlog.

What we'd do differently

No project like this is ever fully clean! Here’s what we’d consider trying to improve if we started again:

• There was no automated eval. Human review was our quality bar, which kept us safe from semantic-drift bugs, but also meant our quality claim rested on reviewers, not a reproducible score. If we scale it further, we’d want to explore building an eval harness.
• The model returned the whole file on every call. Our pipeline synthesizes a diff from that full response via git apply, so we didn't need the model to produce one. But regenerating the whole page burns output tokens on text that hasn't changed. A diff-based approach – where the model returns only the changed lines – would cut most of that back. At a larger scale, that's the optimization we'd reach for.
• Merge-status sync stayed partly manual. Internally, we kept a tracking file for each code as it moved through the batch script: selected, updated, and marked complete once a PR was opened. Then we manually checked that bookkeeping against GitHub. With another run, we’d try to automate that reconciliation as well.

Try it, and tell us how it goes

If you maintain error docs – or, really, any big pile of docs that quietly goes stale – the reusable core here is yours. Grab our prompt at github.com/twilio-labs/docs-ai-buddy, start with the [CUSTOMIZE] sections, and make it your own.

Any issues? I'd genuinely love to hear about them. Open an issue, send feedback, then tell us what worked and what broke.

And of course, go check out the docs this effort produced at twilio.com/docs/api/errors and let us know what you think. Better error docs are one of those quiet things that make a developer's worst day a little less bad. Let's build more of them together!

Maria Bermudez is a Developer Educator at Twilio, where she helps developers get unstuck without pretending the tools that unstick them are magic. Lately that's meant teaching a script and a language model to fix a thousand-plus error-code pages, one human-reviewed diff at a time.