# Do AI crawlers actually read llms.txt? > We logged every self-identified AI crawler hitting this site for 89 days: 10,993 requests from 13 bots. Exactly 33 were for llms.txt. Here is what they fetched instead. Most writing about llms.txt argues from the spec. This argues from a server log. We kept every request to this site for 89 days and counted what the AI crawlers actually asked for. ## The short answer They visit constantly and they almost never ask for the file. Over 89 days (24 July to 22 October 2026) thirteen self-identified AI crawlers made **10,993 requests**. **33** of them were for `/llms.txt`. The same crawlers fetched `/robots.txt` 1,365 times. ## How this was measured, and what it cannot tell you Read the limits before the numbers, because they are real: - **One site.** This one — a free llms.txt validator. A site *about* llms.txt is, if anything, biased toward attracting interest in the file, which makes the result more striking rather than less, but it is still a single sample. - **Self-identified bots only.** Classification is by User-Agent. Anything crawling without saying so is counted as something else, so these are floors, not totals. - **89 days, not forever.** Analytics rows are pruned, so this is a rolling window, not the site's whole history. - **Requests, not understanding.** A fetch proves a file was wanted. It proves nothing about what was done with it. ## Who came CrawlerRequestsLast seen ClaudeBot2,63522 Oct Amazonbot2,08022 Oct ChatGPT-User1,70222 Oct PerplexityBot1,58122 Oct OAI-SearchBot1,43922 Oct GPTBot62322 Oct Meta-ExternalAgent39520 Oct Bytespider22322 Oct YouBot8915 Oct Claude-User8522 Oct Google-Extended7627 Aug CCBot4131 Aug Perplexity-User2427 Aug Nine of the thirteen appeared in the last week of the window. Google-Extended, CCBot and Perplexity-User have not been back since August. ## What they asked for PathRequestsvs llms.txt `/robots.txt`1,36541x `/sitemap.xml`70021x Markdown page versions (`.md`)1594.8x `/llms.txt`33— `/llms-full.txt`240.7x `/llms.json`90.3x `/.well-known/agent-card.json`50.2x The rest, 8,680 requests, was ordinary HTML: the homepage, the guide index, and 58 different guide articles. They are reading the content. They are navigating to it the way crawlers always have — robots.txt for permission, sitemap.xml for the map — and walking straight past the file built for them. ## The one result that points the other way Markdown versions of pages were fetched **159 times across 31 distinct URLs**, roughly five times the traffic of llms.txt itself. The breakdown is not what we expected either: Crawler`.md` fetches Meta-ExternalAgent53 Amazonbot33 ClaudeBot31 OAI-SearchBot30 ChatGPT-User6 PerplexityBot3 Meta-ExternalAgent, a crawler nobody discusses in llms.txt threads, is the single biggest consumer of our Markdown. And Markdown versions are exactly what [spec v2](/guide/llms-txt-v2-what-changed) added link relations for: `rel="alternate" type="text/markdown"` on a page pointing at its own clean rendering. Point crawlers at Markdown and some of them take it. Publish a curated index and they ignore it. ## So should you still publish an llms.txt? Yes, and the honest reason is not traffic. It costs an afternoon, it does not slow anything down, and the moment adoption moves you are already there rather than three months behind. Several documentation platforms generate one automatically now, and coding agents do read them on purpose when a developer hands them the URL — that is real consumption, it just does not look like crawler traffic in a log. But if you are choosing where to spend the next afternoon, this data says spend it on the boring layer first: a correct `robots.txt` that allows the answer engines, a current `sitemap.xml`, and clean pages underneath. That is what the crawlers are actually using. Then publish a Markdown version of your key pages and advertise it with the v2 link relation — on this evidence, more agents will take that than will ever ask for your llms.txt. You can check all of it, including which of these files your site serves and whether the v2 relations are present, with the [validator](/). ## Reproducing this Nothing here needs special tooling. Any access log with a User-Agent column will do: group requests by the crawler names above, then count how many landed on each path. If you run the same count on your own site we would like to see it — the interesting question is whether a site that is *not* about llms.txt sees the same ratio, and one log cannot answer that.