The llms.txt Generator writes a spec-valid file, not a dump.

llms.txt is a proposed markdown file at the root of a website that gives language models a curated index of the site’s most useful pages. The format, proposed by Jeremy Howard of Answer.AI, is deliberately small: one H1 with the site name, which is the only required element, an optional blockquote summary, optional prose, and then H2 sections whose entries are markdown links with optional notes. It is a suggestion to models, not a control, and no engine is obliged to read it.

No. Google’s John Mueller compared llms.txt to the keywords meta tag: a site owner’s claim about their own content that engines can simply verify by reading the site. He also noted that server logs show the AI services do not even fetch the file. Nothing published since has changed that position, so generate an llms.txt for the cheap upside if you want one, but do not expect it to move Google rankings or AI Overviews.

None commits to it, and the measured traffic is close to zero. Ahrefs studied 137,210 domains in 2026: 28% publish an llms.txt, and 97% of those files received no requests at all in the month measured. Of the tiny fraction of files that were fetched, most requests came from SEO audit tools, and AI retrieval bots were about 1%. OpenAI, Anthropic and Perplexity all publish llms.txt files for their own documentation, but none of the three documents ever reading yours.

Because the demand is real, the cost is nothing, and an honest generator beats a hyped one. More than a quarter of studied domains publish the file, standards need adoption before consumption, and the file doubles as a clean, human-readable map of your best pages that coding agents and documentation tools already use. What RankX AI will not do is sell llms.txt as an AI visibility tactic, because the evidence says it is not one. The generator exists so that if you want the file, you get a correct, curated one in seconds.

The spec is stricter than most generators respect. A valid file has exactly one H1 naming the site, and that H1 is the only required block. After it come, in order, an optional blockquote summary, optional prose with no further headings, and then H2 sections containing markdown lists where each entry is a link in the form of a name and URL, optionally followed by a colon and a note. No H3s, no tables, no HTML. This generator validates its own output against those rules before handing it to you.

An H2 section literally named Optional has reserved meaning by convention: it marks links an AI agent can skip when it is short of context space. Everything above it is what you most want a model to read; everything under Optional is the long tail. That is also why order matters everywhere else in the file, because a model that truncates reads from the top.

llms-full.txt is a documentation-platform convention, not part of the llms.txt specification, which never mentions it. Platforms such as Mintlify and Fumadocs generate it as the entire documentation corpus concatenated into one file, so an agent can ingest everything in a single fetch. It makes sense for docs sites and very little sense for a marketing site, so this generator deliberately produces only llms.txt and leaves llms-full.txt to the platforms that build it from their own source.

No. A claim circulates that Anthropic honours a Commercial-Use directive inside llms.txt, sometimes expanded into a whole invented directive set with Training-Data and Citation-Required fields. It appears in no version of the specification and in no vendor documentation from Anthropic or anyone else, and it traces back to a single SEO blog post that cites no source. Be careful: AI search summaries repeat it as fact. If you want to control training use, robots.txt tokens are the documented mechanism.

Because curation is the entire point of the format. An llms.txt that mirrors your whole sitemap is just a worse sitemap, and models that truncate long context lose whatever you put at the bottom. The generator ranks pages, keeps roughly the best fifty, orders sections by importance and writes each link’s note from the page’s own meta description, so what survives truncation is what you would have chosen anyway. Dumping every URL is the measured weakness of most competing generators.

The generator reads your sitemap, or discovers it from robots.txt, fetches up to 100 pages politely at one request at a time while honouring your robots.txt, and extracts each page’s title and meta description. It then groups pages into sections, ranks them, and writes a spec-valid file with your homepage’s description as the blockquote summary. You can edit anything before you publish it, and repeat runs inside 24 hours return the cached result instantly.

Serve it at the root of your site as /llms.txt, next to robots.txt. The spec also allows files at subpaths, such as /docs/llms.txt, and says agents should use the most specific file that covers a URL, which is useful when your documentation deserves its own index. Serve it as plain text or markdown, keep it in version control like any other content, and update it when your key pages change.

They point in opposite directions. robots.txt says what crawlers may not fetch, is a published standard, and the major AI vendors document how their bots treat it. llms.txt says what models should read first, is a proposal with no committed consumers, and carries no enforcement at all. A site can sensibly have both: robots.txt as the access policy, llms.txt as the curated index. Our AI Crawler Access Checker covers the first; this generator covers the second.