Announcing Twilio Docs Support for llms.txt and Markdown

Announcing Twilio Docs Support for llms.txt and Markdown

The Twilio documentation platform now supports your LLM workflows by providing an llms.txt file and Markdown versions of every docs page.

Add the .md file extension to any docs URL to get the LLM-optimized Markdown or use the Page tools links above the table of contents ( On this page) to view or copy the Markdown directly to your clipboard.

Why Markdown matters

HTML is noisy. It includes markup to create visual navigation, tabbed interfaces with conditionally hidden content, and other features meant for humans operating through a screen. Not only is this markup unnecessary for an LLM, it consumes tokens and may cause confusion.

Markdown was created by John Gruber as a "text-to-HTML conversion tool for web writers" in 2004. Unlike HTML, Markdown uses minimal plaintext formatting to shape documents, allowing authors to create structure with fewer special symbols and less visual clutter.

For a clear difference, compare the HTML and Markdown required to format a bulleted (unordered) list.

HTML unordered list

Markdown unordered list

Markdown's minimalist approach happens to work well for LLMs. It supplies all the necessary semantic structure in a format that is cleaner than HTML, making it easier to parse and chunk. Ultimately, Markdown provides all the context, including multi-language code samples, with fewer tokens, leaving your context window open for instructions and other additional data.

Markdown from the source

You may be wondering how our Markdown is optimized or different from the results you get by scraping with Firecrawl or using what's available on Context7. Though other tools can scrape our docs and generate usable results, we can modify the output at the source.

In 2024, we migrated Twilio's documentation to our own docs as code (DaC) platform. We built the platform on Next.js, and our content source files are written and maintained in MDX. This means the underlying source data is largely Markdown to begin with. We build JSX components to supplement Markdown with everything from basic callout blocks to complex code and API reference generators that consume our OpenAPI specs.

Rather than scrape our HTML, we created a separate build pipeline based on our existing remark plugins. This allows us to make thoughtful transformations for each component, and we can update those transformations based on feedback. For example, if you visit the Markdown version of an API reference page, you'll notice that the Properties table is rendered as a JSON code block rather than a Markdown table. Knowing that the Markdown is meant for a machine, we opted to provide something machine readable.

Because we do this at build time, we can also keep the Markdown up to date. You won't arrive at the .md version of a page to find it missing critical updates that are available in the HTML version.

What is llms.txt

The llms.txt proposed standard is designed to help LLMs understand a website by providing structured information and curated links to Markdown page content. Think of llms.txt like a curated sitemap intended specifically for LLMs, similar to how robots.txt guides web crawlers. You can seed your crawler with https://www.twilio.com/docs/llms.txt to fetch the most relevant Markdown files we provide, ensuring it ingests the cleanest Twilio documentation for your LLM.

Upgrade your Twillio and LLM workflow

As a builder integrating Twilio APIs and services into your products, you can replace any Twilio HTML docs in your prompts or context bundles with a .md URL today to upgrade your Twilio experience immediately.

As an LLM tool provider, you can now ingest a standardized Markdown content feed discoverable via llms.txt, which will reduce your crawl and parsing overhead, and improve your retrieval precision.

Keep your models well-fed (and token counts low)!

Thank you to Twilio Developer Education’s Docs Platform team for making this feature a reality.

Wade Christensen is a technology professional with a background in developer education and developer tools. At Twilio, he is the Product Manager for Twilio's documentation platform. You can reach him at wchristensen [at] twilio.com