llms.txt

What does the llms.txt specification actually require?

Less than most generators produce. The specification, proposed by Jeremy Howard on 3 September 2024, requires exactly one element: an H1 with the project name. Everything else is optional, including the blockquote summary, the H2-delimited lists of links with notes, and the conventional Optional section marking material an agent may skip. The file lives at /llms.txt, and a file at a subpath covers the URLs beneath it.

It is worth reading the source rather than a tool’s interpretation of it, because the specification claims no search or citation benefit anywhere. That claim was added by the industry that grew around it.

Does anything actually read llms.txt?

Almost nothing, and this is the best-measured question in the whole area. Ahrefs looked at 137,210 domains in May 2026 and found that 97% of llms.txt files received zero traffic that month. Of the requests that did arrive, 96% came from bots, AI bots accounted for 19.5% of fetches to files with any traffic, and 12% of fetches were, in the study’s phrase, “the industry studying itself”.

Its authors’ conclusion is quotable and blunt: “If your goal is showing up in ChatGPT, Perplexity, or AI Overviews, an llms.txt file is largely decoration.”

Should you publish one anyway?

Yes, if it costs you nothing and no, if it costs you anything. Generated from the same source as your sitemap it is a few minutes of work, it is genuinely useful to developers pointing an agent at your documentation, and Chrome’s Lighthouse now audits for its presence. Hand-maintained, it is a second content inventory that will silently drift out of date.

What it is not is an AI visibility tactic, and treating it as one is a useful tell. An audit or an agency that leads with llms.txt is describing a file almost nothing fetches, which says something about the rest of the recommendations. The adjacent idea with more behind it is serving markdown versions of your pages, generated from the same source as the HTML. Nothing documents requesting them either, and at least the surface is the content itself rather than a list of links to it.

This site publishes both, from one read, which is the only arrangement where the cost of being wrong about either is zero. The rule that separates the two cases is whether the artefact is generated or maintained. Anything generated from the source that already renders the page costs one build step and cannot drift; anything a person has to remember to update is a liability with a publication date on it.

Generating one from your sitemap?

Related terms

• AI CrawlerAn AI crawler is an automated fetcher run by an AI company to collect web pages.
• Robots.txtRobots.txt is a file at a site’s root telling automated crawlers which paths they may fetch.
• XML SitemapAn XML sitemap is a file listing the URLs on a site that you want search engines to know about, optionally with the date each was last modified.
• Atomic ContentAtomic content is the practice of writing in self-contained units, each covering one idea completely enough to be understood alone.