Guide

Do AI crawlers actually read llms.txt?

Published: 2026-10-22

Most writing about llms.txt argues from the spec. This argues from a server log. We kept every request to this site for 89 days and counted what the AI crawlers actually asked for.

The short answer

They visit constantly and they almost never ask for the file. Over 89 days (24 July to 22 October 2026) thirteen self-identified AI crawlers made 10,993 requests. 33 of them were for /llms.txt.

The same crawlers fetched /robots.txt 1,365 times.

How this was measured, and what it cannot tell you

Read the limits before the numbers, because they are real:

• One site. This one — a free llms.txt validator. A site about llms.txt is, if anything, biased toward attracting interest in the file, which makes the result more striking rather than less, but it is still a single sample.
• Self-identified bots only. Classification is by User-Agent. Anything crawling without saying so is counted as something else, so these are floors, not totals.
• 89 days, not forever. Analytics rows are pruned, so this is a rolling window, not the site's whole history.
• Requests, not understanding. A fetch proves a file was wanted. It proves nothing about what was done with it.

Who came

Nine of the thirteen appeared in the last week of the window. Google-Extended, CCBot and Perplexity-User have not been back since August.

What they asked for

The rest, 8,680 requests, was ordinary HTML: the homepage, the guide index, and 58 different guide articles. They are reading the content. They are navigating to it the way crawlers always have — robots.txt for permission, sitemap.xml for the map — and walking straight past the file built for them.

The one result that points the other way

Markdown versions of pages were fetched 159 times across 31 distinct URLs, roughly five times the traffic of llms.txt itself. The breakdown is not what we expected either:

Meta-ExternalAgent, a crawler nobody discusses in llms.txt threads, is the single biggest consumer of our Markdown. And Markdown versions are exactly what spec v2 added link relations for: rel="alternate" type="text/markdown" on a page pointing at its own clean rendering. Point crawlers at Markdown and some of them take it. Publish a curated index and they ignore it.

So should you still publish an llms.txt?

Yes, and the honest reason is not traffic.

It costs an afternoon, it does not slow anything down, and the moment adoption moves you are already there rather than three months behind. Several documentation platforms generate one automatically now, and coding agents do read them on purpose when a developer hands them the URL — that is real consumption, it just does not look like crawler traffic in a log.

But if you are choosing where to spend the next afternoon, this data says spend it on the boring layer first: a correct robots.txt that allows the answer engines, a current sitemap.xml, and clean pages underneath. That is what the crawlers are actually using. Then publish a Markdown version of your key pages and advertise it with the v2 link relation — on this evidence, more agents will take that than will ever ask for your llms.txt.

You can check all of it, including which of these files your site serves and whether the v2 relations are present, with the validator.

Reproducing this

Nothing here needs special tooling. Any access log with a User-Agent column will do: group requests by the crawler names above, then count how many landed on each path. If you run the same count on your own site we would like to see it — the interesting question is whether a site that is not about llms.txt sees the same ratio, and one log cannot answer that.

Keep reading

• Which AI crawlers read llms.txt?A practical rundown of the AI crawlers and answer engines — GPTBot, ClaudeBot, PerplexityBot and more — and how llms.txt and robots.txt control what they see.
• llms.txt vs robots.txt vs sitemap.xmlThree root files, three different jobs. What each one controls, why llms.txt is not access control, and the robots.txt rules to write for AI crawlers.

Validate your llms.txt →