llms.txt vs robots.txt vs sitemap.xml: what each actually does for AI.
Three plain-text files at the root of your site. Two are standards every crawler reads. The third is a proposal with a lot of enthusiasm and, so far, very little evidence.
We publish an llms.txt. We also tell clients not to expect it to do anything for AI search. Both are true, and here is why.
What each file does
| File | Purpose | Who reads it | Effect on AI visibility |
|---|---|---|---|
| robots.txt | Allows or blocks crawlers by path | Every major search and AI crawler | Direct: blocking a search bot can remove you from its answers |
| sitemap.xml | Lists URLs and last-modified dates | Search engines; AI search crawlers that discover via search indexes | Indirect: faster, fuller indexing, which AI search depends on |
| llms.txt | A Markdown summary of the site for language models | Some coding agents and documentation tools | No measured effect so far |
The evidence on llms.txt
- Adoption is real but uneven. Trackers put it at about 8.7% of the Tranco top 1,000 (June 2026) and about 10% in a 300,000-domain study, far higher among developer-tool sites.
- The big AI crawlers don't ask for it. A server-log study of about 900 domains, September 2025 to April 2026, logged 1,227 llms.txt requests in total and none from GPTBot, ClaudeBot, PerplexityBot or Google-Extended. About 88% of sites got no requests at all.
- No citation lift. The 300,000-domain study found no measurable link between having llms.txt and how often a site was cited.
- Google's position. Google's AI features documentation says you don't need new machine-readable files or AI text files to appear in AI Overviews or AI Mode.
So why do we publish one?
Because it costs fifteen minutes and some readers do use it: coding agents, documentation tools and anyone who points a model at our site directly. Our llms.txt is generated from the same data as the sitemap, so it can't go stale. We just don't count it as GEO work.
Where to spend the time instead
- robots.txt: make sure AI search and user bots are allowed. See which bots to allow.
- sitemap.xml: accurate lastmod dates, submitted to Google and Bing, plus IndexNow.
- The pages themselves: server-rendered, answer-first, with sources. See how AI engines choose sources.
Our free robots.txt AI generator builds the file that actually matters, and the AI crawler checker confirms bots can reach you.
Questions people ask
Does llms.txt help with AI search?
There is no evidence it does yet. Log studies found the major AI crawlers don't request it, a large study found no citation lift, and Google says no AI text files are needed.
What is the difference between llms.txt and robots.txt?
robots.txt controls which crawlers may fetch which pages and is honoured by major crawlers. llms.txt is an optional Markdown summary of a site for language models, with no access-control role.
Should I still create an llms.txt file?
If coding agents, documentation tools or AI pipelines read your content, it is cheap and useful. Don't treat it as an SEO or AI-ranking lever.
Want this mapped onto your product?
See how our GEO work runs, or send your store link and a founder will reply personally.
