llms.txt vs robots.txt vs sitemap.xml
Published:
All three live at your site root and all three are plain text — but they solve different problems. Mixing them up is the most common llms.txt mistake.
The one-line summary
- robots.txt — controls crawler access: what bots may or may not fetch.
- sitemap.xml — lists every indexable page for search engines.
- llms.txt — curates a shortlist of high-signal pages for large language models at inference time.
robots.txt is about permission
robots.txt tells crawlers which paths they're allowed to request. It is an access-control convention. It does not curate or rank content, and it isn't written for language models specifically.
sitemap.xml is about coverage
A sitemap aims to be exhaustive: it maps all the URLs you want indexed so search engines don't miss anything. It's machine-oriented XML, not a human-readable summary.
llms.txt is about curation
llms.txt is the opposite of exhaustive. It's a hand-picked, Markdown list of the few pages that best help a model understand and use your site — with a short description on each. It exists for inference (helping when a user asks an AI for help), not for training datasets and not for access control.
They complement each other
You don't choose between them — a strong setup has all three. robots.txt governs access, sitemap.xml ensures coverage, and llms.txt curates signal for AI. Once you've written yours, run it through the validator and read how to create an llms.txt file for the format details.