Markdown for agents and Content Signals: serving AI efficiently, and on your terms.
Two small, separate ideas from the agent-readiness world: give agents a lighter version of your page when they ask, and tell them in robots.txt what they may use it for.
An agent reading your page pays for every token of navigation, scripts and markup before it reaches the content. And it has no way to know what you are happy for it to do with that content. These two ideas address both.
Markdown for agents
The idea is HTTP content negotiation: a client that prefers Markdown sends Accept: text/markdown, and the site returns a Markdown version of the same page. Cloudflare launched "Markdown for Agents" in February 2026 as a beta for Pro, Business and Enterprise plans: it converts the HTML at the edge and adds an x-markdown-tokens header with an estimated token count.
- Why it matters: Cloudflare's launch post measured itself at 16,180 tokens as HTML and 3,150 as Markdown, about an 80% cut.
- Who asks: Cloudflare says coding agents such as Claude Code and OpenCode already send Markdown Accept headers.
- Adoption: 3.9% of the 200,000 top domains Cloudflare scanned handle the negotiation correctly.
The caveat: Google's guide to generative AI features says you don't need Markdown, AI text files or special markup to appear in Google Search, including AI Overviews, because Google Search doesn't use them. Markdown helps other agents and tools, not Google. If you serve it, keep it identical in substance to the HTML; Google's John Mueller has criticised separate bot-only pages.
Content Signals
Cloudflare published the Content Signals Policy in September 2025. It adds a Content-Signal line to robots.txt with three preferences:
| Signal | Covers |
|---|---|
| search | Building a search index and showing links or snippets |
| ai-input | Using the content as input to AI-generated answers |
| ai-train | Training AI models |
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /Cloudflare applies search=yes, ai-train=no by default for customers using its managed robots.txt, and deliberately leaves ai-input unset. A missing signal means no stated preference, not permission or refusal.
What we recommend
- Content Signals: add them. They cost one line and state your intent clearly. Most businesses that want to be cited should say yes to search and ai-input; ai-train is a separate business decision. Our robots.txt generator handles the bot rules around it.
- Markdown: optional. Worth it for documentation, help centers and API references, where agents and developers read heavily. Less important for marketing pages.
- Neither replaces access. If a bot can't reach you, signals and Markdown don't matter. Check with our AI agent log analyzer.
Questions people ask
What is Markdown for Agents?
Serving a Markdown version of a page when a client sends Accept: text/markdown. Cloudflare offers it as an edge feature that converts HTML automatically and reports a token estimate.
What are Content Signals in robots.txt?
A Content-Signal line declaring yes or no for three uses: search, ai-input (use in AI answers) and ai-train. They express preferences; they don't technically block anything.
Will serving Markdown to bots help or hurt my SEO?
Neither, for Google: Google says Google Search doesn't use Markdown or AI text files. It can help other agents and tools read your pages. Keep it the same content as the HTML and avoid bot-only pages.
Sources
Updated
Want this mapped onto your product?
See how our AAO work runs, or send your store link and a founder will reply personally.
