Which AI crawlers read llms.txt?
Published:
llms.txt is a young, voluntary convention, so no crawler is strictly required to read it — but a clean file plus clean pages helps any model that does. It also pairs with the file that actually decides access: robots.txt. Think of robots.txt as the gate and llms.txt as the curated map.
Two kinds of AI crawler
It helps to split AI bots into two groups, because you usually treat them differently:
- Answer / search crawlers — fetch pages to answer a user's question right now:
OAI-SearchBotandChatGPT-User(OpenAI),Claude-SearchBotandClaude-User(Anthropic),PerplexityBotandPerplexity-User. These drive AI visibility — you generally want them. - Training crawlers — collect data to train models:
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBot,Bytespider,Meta-ExternalAgent. Whether to allow these is a separate, opt-out choice.
Which of them read llms.txt today?
Honestly: adoption is still early. llms.txt is an emerging standard with growing support among AI developer tools and agent frameworks, but most large crawlers do not yet fetch it automatically. The value is in being ready — a correct, curated file costs little and positions you as the standard matures, while the clean pages it points to help models right now.
Access is decided in robots.txt, not llms.txt
llms.txt never blocks or grants anything — it only curates. To control what AI crawlers may fetch, use robots.txt. A visibility-friendly setup allows the answer crawlers and makes a deliberate choice about the training ones:
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: /
See llms.txt vs robots.txt for how the two files divide the work.
What to do
- Allow the answer/search crawlers above so AI tools can cite you.
- Decide training opt-outs (
GPTBot,Google-Extended) deliberately — blocking them does not affect AI search visibility. - Publish a curated llms.txt, then validate it so every linked page actually resolves.