What Is llms.txt, and Do You Need One?

What is llms.txt?

llms.txt is a proposed standard from Jeremy Howard of Answer.AI, September 2024: a Markdown file at your site's root that gives AI systems a curated, machine-readable summary of the site, what it is, which pages matter, where the clean versions live. llms.txt is designed to help large language models spend a finite context window well, and the llms.txt specification also defines a companion, llms-full.txt, which serves the entire content corpus as one file.

The idea is reasonable on its face. Assistants work from fetched web content, fetching is expensive, and a site-provided index could save everyone the crawl, which is why developer docs platforms and API references were the earliest adopters: a docs page maps naturally onto a curated link list. The question that matters is not whether the llms.txt standard is elegant; it is whether anything reads the file.

Does anything actually read it?

Mostly, no, and this is measured rather than argued. Ahrefs analysed server logs across 137,000 domains and found 97 percent of llms.txt files received zero requests. Of the requests that did arrive, around 1 percent came from AI retrieval bots; most were SEO audit tools checking whether the file exists, which is a loop with no user in it.

The absence goes to the top of the stack: OpenAI, Anthropic, Google and Perplexity each document their crawlers in detail, and none documents consuming llms.txt, so neither ChatGPT, Claude nor Gemini can be shown to know about llms.txt at retrieval time. Google has said directly that the file is not required and confers no effect. A separate 300,000-domain comparison found no citation difference between sites with and without one. Adoption grew almost ninefold in a year anyway, which says more about how this field spreads advice than about the file.

How does llms.txt compare to robots.txt and sitemap.xml?

The three files sound alike and do opposite jobs. A robots.txt file excludes: it tells compliant crawlers what not to fetch, and search engine optimization has leaned on it for three decades. A sitemap.xml enumerates: every URL, no judgement, so nothing gets missed. An llms.txt file curates: the pages worth an AI system's attention, with a short description each. Exclusion, enumeration, curation.

The differences that matter are enforcement and adoption. Standards like robots.txt work because consumers exist: crawlers honour robots.txt imperfectly but measurably, and sitemaps demonstrably help search engines find URLs. The existing standards earned their place; llms.txt has a syntax and a website but, so far, no documented consumer among the major engines or AI models. Structured data sits in the same drawer: like schema markup, an llms.txt file can only mirror content that must already stand on its own, and neither buys AI citations in controlled tests.

Why publish one anyway?

Because the cost can be zero, and at zero cost even a small option is worth holding. This site serves an llms.txt and a full llms-full.txt corpus, generated from the same constants and collections that render the visible pages, so the file cannot drift from the site and its maintenance cost after setup is nothing. If an engine starts consuming the convention, we are already legible to it; if none ever does, we spent nothing that mattered. Sites that use llms.txt today do it for that option value, not for a measured effect.

The same logic in reverse is a warning sign worth naming: an SEO expert whose audit leads with your missing llms.txt is leading with the cheapest, least consequential box on the checklist. The retrieval mechanics that decide visibility, rendering, crawler access, extractable structure, are covered in the GEO guide; none of them lives in this file.

How do you create an llms.txt file?

The basic structure comes straight from the llms.txt proposal, and any Markdown tool can parse it: one h1 heading with the site name, a blockquote carrying a concise summary, then h2 headers for each section, Home, Docs, Blog, whatever fits, each holding a list of URLs with a short description per link. Save it as a plain text file, upload it to the site root so it resolves at /llms.txt, and serve it as text; no plugin, no build step, one file in a public directory.

The free llms.txt Generator reads your site and drafts the file: it finds the pages worth listing, writes the descriptions from what the pages actually say, and outputs the structured format ready to serve at the root. It runs without an account. Whichever way you create llms.txt content, four rules keep it useful:

• Lead with what the site is, in one paragraph a machine can quote.
• Curate the important, high-value content that answers real questions, not every URL; the sitemap already handles exhaustive.
• Write honest one-line descriptions of your best content. The file's only conceivable reader is a system deciding what to fetch; a description that oversells earns a fetch that disappoints.
• Generate it from your content source if you can, so it stays up-to-date content rather than a snapshot. A hand-written file is out of date at the first publish after it.

The adjacent idea with more substance: Markdown twins

A related convention has real infrastructure behind it: serving Markdown versions of each page, either at a .md URL or through an Accept header, which Cloudflare now supports at the edge. This site does that too, every HTML page has a Markdown twin generated from the same read, with the HTML kept as the indexable surface. The honest caveat is the same shape as before: no engine has stated it requests Markdown. Engines already convert your HTML to Markdown themselves, which is the actual lesson of the whole area: the durable investment is clean, semantic, server-rendered HTML with nothing meaningful hidden behind JavaScript, because that is the input every pipeline shares. Whether your pages pass that bar is checkable with the site audit in Website Audit.

Sources

• Ahrefs, llms.txt server-log study, 137K domains, checked 20 Aug 2026
• llms.txt proposal, Answer.AI, checked 20 Aug 2026
• Google, AI features and your website, checked 20 Aug 2026
• OpenAI, Anthropic and Perplexity crawler documentation (no llms.txt consumer documented), checked 20 Aug 2026