What Is llms.txt, and Do You Need One?
What the llms.txt file is, what the server-log evidence says about who reads it, why we publish one anyway, and how to create yours in a minute.

On this page 10 sections
- What is llms.txt?
- What does an llms.txt file look like?
- How does it fit with existing standards like robots.txt and sitemap.xml?
- Does anything actually read llms.txt?
- Does llms.txt help SEO or AI visibility?
- Should you use llms.txt on your site?
- How do you create an llms.txt file?
- What should a store put in the file?
- What are Markdown versions of pages, and do they matter more?
- How do you check the file and the pages it points to?
Short answer: llms.txt is a proposed Markdown file at a website's root that gives AI agents and large language models a short, curated summary of the site and links to its most useful pages. It is optional and it is not a ranking signal: Google Search ignores it, and no major AI search engine documents reading it. Publish one when you can keep it accurate at little cost.
What is llms.txt?
An llms.txt file is a plain text file, written in Markdown, that sits in a website's root directory and tells AI systems what the site is and which pages matter most. Jeremy Howard of Answer.AI proposed the format in September 2024. The aim is to help large language models spend a limited context window on a site's best content instead of trying to parse navigation, scripts and ads.
The file is machine-readable and human readable at once. It holds a title, a concise summary and lists of links, each with a one-line description. An agent reads the summary first, then fetches only the links it needs.
Version 2 of the proposal, published in August 2026, widened where the file can live. A site can serve it at the root, or in a subdirectory such as /docs/, where it covers only the pages under that path. Where two files apply, an agent should use the most specific one.
Why do doc sites use it most?
Developer docs are the heaviest users. Coding agents follow these files to find API references, tutorials and README files, and documentation platforms such as Mintlify and GitBook generate one automatically for every doc page they host. On a docs site, an llms.txt might point an agent straight at the API reference it needs.
OpenAI, Anthropic and Google's Gemini team each publish one for their own developer documentation.
Many docs sites also publish llms-full.txt, one file that joins every documentation page into a single long Markdown document. That convention grew up around the proposal, and the llms.txt specification itself does not define it.
What does an llms.txt file look like?
The basic structure has one required part, an H1 with the site name, and four optional parts in a fixed order. That structured format is what lets a parser read the file with ordinary code as well as with an LLM.
Part | Markdown syntax | Required? |
|---|---|---|
Site or project name | One H1 heading | Yes, the only required section |
Summary | A blockquote with the key facts a reader needs | No |
Notes | Paragraphs or lists, with no headings | No |
File lists | H2 headers, each over a list of links with an optional note after a colon | No |
Optional | An H2 named Optional, for links an agent can skip when context is short | No |
Here is a short example for an invented store, Juniper & Co, one line at a time:
# Juniper & Co> Juniper & Co sells handmade stoneware tableware and ships across the UK and EU.## Shop- [Dinner plates](https://example.com/collections/plates): Stoneware plates in six glazes, dishwasher safe## Help- [Delivery and returns](https://example.com/pages/delivery): Delivery times, costs and the 30-day return window## Optional- [Our story](https://example.com/pages/about)
Each link should point at clean web content. The proposal suggests a Markdown copy of each linked HTML page, so an agent gets the text without the markup around it.
How does it fit with existing standards like robots.txt and sitemap.xml?
The three root files sound alike and do different jobs: robots.txt excludes, sitemap.xml enumerates, and llms.txt curates. Search engine optimization has leaned on the first two web standards for decades, because crawlers read them.
- robots.txt
Communicates crawler directives.
- sitemap.xml
Lists URLs for discovery.
- llms.txt
Offers a proposed content guide.
A content guide does not replace crawl rules or establish that a system will use it.
File | Its job | Status | Who reads it |
|---|---|---|---|
robots.txt | Tells compliant crawlers which paths not to fetch | Internet standard, RFC 9309 | Search and AI crawlers, before they fetch |
sitemap.xml | Lists every URL you want indexed | Sitemaps protocol | Search engine crawlers |
llms.txt | Points agents at your best pages, with a note on each | Community proposal | Agents and LLM tools that choose to read it |
The differences that matter are enforcement and adoption. Standards like robots.txt work because consumers exist: crawlers honour a robots.txt file imperfectly but measurably, and sitemaps demonstrably help search engines find URLs. The newer file has a specification and a website, but no documented consumer among the major AI search engines.
The proposal says llms.txt is designed to coexist with robots.txt and sitemap.xml, not replace them. It cannot block a crawler, and it cannot submit a page for indexing. To control which bots can use your content, edit robots.txt and test it with the AI Crawler Access Checker.
Structured data sits in the same drawer. Like schema markup, a curated index can only mirror content that must already stand on its own, and neither has been shown to buy AI citations.
Does anything actually read llms.txt?
Mostly, no, and this is measured rather than argued. Ahrefs analysed server logs across 137,000 domains and found 97 percent of these files received zero requests. Of the requests that did arrive, around 1 percent came from AI retrieval bots.
The largest single group of requests, 21.7 percent, came from SEO audit tools. In total, 77 percent of the bots fetching the file were not AI tools at all. Adoption spread anyway, to 28 percent of domains in the Ahrefs study.
The absence goes to the top of the stack. OpenAI, Anthropic, Google and Perplexity each document their crawlers, and none documents consuming the file. So the AI models behind ChatGPT, Claude, Gemini and Perplexity cannot be shown to read yours at retrieval time.
What does the Lighthouse audit check?
Chrome's Lighthouse now looks for the file in its agentic browsing checks. The audit flags a server error when it fetches the address, and it marks a missing file as not applicable, because the format is optional. A site without one loses no Lighthouse score.
Some of the growth is automatic rather than chosen. Our survey of 359 live WordPress sites in September 2026 found 18.6 percent of the 354 that could be checked serving one, and 61 of those 66 files sat on a site running a detectable SEO plugin. The measurement sits in what WordPress sites actually serve to AI crawlers.
To see whether GPTBot, ClaudeBot or any other bot requests your own file, read your server logs. AI bot tracking shows how to find those requests and verify that each bot is genuine.
Does llms.txt help SEO or AI visibility?
No study shows that llms.txt improves rankings or citations, and nothing shows the file helps AI crawlers find pages. Google's guide to optimizing for generative AI features says Google Search ignores these files, so one neither helps nor harms your Google rankings or AI Overviews. SE Ranking's analysis of nearly 300,000 domains found no effect on how often a site was cited.
The proposal itself expects the file to be read on demand, when an agent that reads content from websites needs information while helping a user. That is a different job from building a search index. An agent with a browsing tool can follow it, and a crawler building a search index has no documented reason to.
AI visibility is decided earlier in the pipeline. A page has to be fetchable, rendered on the server and clear enough to quote. The GEO guide covers those mechanics, and none of them lives in this file.
From RankX AICreate a file only after checking the useful pages.Before spending more time on the file, check that the public product and category pages themselves are accessible, useful and linked.Use the llms.txt generatorShould you use llms.txt on your site?
Publish an llms.txt when you can keep it accurate at little cost. The decision depends on who reads your content and how you would maintain it.
Your site | Should you publish one? |
|---|---|
Developer docs or an API | Yes. Coding agents follow these files, and docs platforms generate them |
An online store | Optional. List collections, buying guides and policies, and generate it from your content source |
A small business site | Optional. Ten good links and a one-paragraph summary are enough |
A site with crawl or rendering problems | Fix those first. A reading list cannot help pages that crawlers cannot read |
Why we publish one anyway
The cost can be zero, and at zero cost even a small option is worth holding. This site serves an llms.txt and an llms-full.txt, generated from the same data that renders the visible pages, so neither can drift from the site. If an engine starts consuming the convention, the site is already readable to it.
The same logic in reverse is a warning sign. An SEO expert whose report leads with your missing file is leading with the cheapest, least consequential box on the checklist. Rendering, crawler access and extractable structure come first.
How do you create an llms.txt file?
Write the Markdown structure above, save it as a text file named llms.txt and serve it from your site's root. For a small site, the whole job takes a few minutes.
Write the summary
One H1 with the site name, then a blockquote that says what the site is in a sentence a machine can quote.
Curate the links
Pick the high-value content that answers real questions, group it under H2 headers, and add a short description after each link.
Serve it from the root
Upload the file so it resolves at your domain root and returns as plain text, next to robots.txt.
Test it
Give an AI assistant only this file and ask it questions about your site. Wrong answers point to a missing link or a vague description.
Keep it current
Regenerate it when key pages change, so it never points agents at a moved or deleted page.
The free llms.txt Generator does the first three steps for you. It reads your sitemap, keeps about fifty pages, writes each note from your own meta descriptions and checks the result against the specification, with no account needed.
On WordPress, Yoast SEO and AIOSEO can generate the file. The free RankX AI WordPress plugin serves llms.txt, agents.md and ai.txt from the site's root without writing anything to disk, and it leaves a file another plugin already serves in place. On Shopify, a theme template named llms.txt.liquid serves your own version.
What should a store put in the file?
A store's index should list only the important content your team is willing to maintain: the store overview, main collections, buying and size guides, and delivery, returns and contact pages. Keeping that list short is what makes it up-to-date content rather than a stale snapshot.
Leave out account, cart, checkout and unstable filtered URLs, and never list private customer information. Check every listed destination after a route change, because a stale link sends an agent to a 404.
The file is the last step, not the first. Make sure the product and category pages themselves are accessible, useful and linked, because a neat Markdown index cannot repair stale availability, contradictory specifications or a broken product URL.
What are Markdown versions of pages, and do they matter more?
A Markdown version of a page is a clean text copy served at a .md URL or through an Accept header, and it has more infrastructure behind it than a curated index. Cloudflare can convert pages to Markdown at the edge, and the proposal recommends a Markdown copy for every page an agent might need.
- Public page
Preserve the full human-readable article.
- Markdown twin
Expose a clean version where supported.
- Discoverability
Link the alternate representation clearly.
A clean representation is useful only if it retains the article meaning.
This site serves Markdown copies of its articles and docs, with the HTML kept as the indexable page. The RankX AI WordPress plugin can publish a Markdown copy of every page once you switch it on, held out of search results. No engine has stated that it requests Markdown, though.
Engines already convert your HTML to Markdown themselves. The durable investment is clean, semantic, server-rendered HTML with nothing meaningful hidden behind JavaScript, because that is the input every pipeline shares. A Website Audit checks whether your pages pass that bar.
How do you check the file and the pages it points to?
Check the file, then check what crawlers can reach. Open the address in a browser and confirm it returns plain text with a 200 status, an H1 first and working links.
Then test the pages behind it. The AI Readiness Score checks a page against the signals AI systems read. RankX AI also re-reads your robots.txt, your llms.txt and your homepage every day on every account and reports when crawler access gets worse.
Next, check what crawlers can retrieve or prioritise SEO and GEO work.
Questions about AI Search Essentials
Does llms.txt help SEO or AI citations?
No ranking or citation benefit has been measured. Google says Google Search ignores these files, and SE Ranking's study of nearly 300,000 domains found no effect on citation frequency. Treat the file as a reading aid for agents and put the SEO work into the pages it links to.
Is llms.txt mandatory?
No. The proposal is optional, and Google does not require any special AI text file for AI Overviews or AI Mode. Chrome Lighthouse marks a missing file as not applicable rather than as a failure.
Is there an official llms.txt standard?
There is a published specification at llmstxt.org, but it is a community proposal, not an internet standard like robots.txt, which is RFC 9309. Jeremy Howard of Answer.AI wrote it in September 2024 and revised it as version 2 in August 2026.
Should the file list every page on the site?
No. The proposal asks for a curated index, not a sitemap dump: the pages that answer real questions, each with one honest line of description. Your sitemap.xml already covers every URL.
Should I create one if my site is small?
You can, and a small site is the easiest case: a one-paragraph summary and ten good links take minutes. Do it after the basics, meaning crawlable pages, a working sitemap and a robots.txt that lets AI search crawlers in. The file adds little if the pages behind it are thin or blocked.
How do I create llms.txt in WordPress?
Yoast SEO and AIOSEO can generate the file, and the free RankX AI WordPress plugin serves it from your site's root without writing anything to disk. The RankX AI plugin is a free download inside RankX AI, and it leaves a file another plugin already serves in place. Check that only one tool answers at the address.
Sources
- Answer.AI (Jeremy Howard), the /llms.txt proposal, v2: format, subpath files, the Optional section, labs that publish their own files, integrations (opens in a new tab) Checked 2026-10-26.
- Ahrefs, server-log study of 137K domains: 97 percent of files never requested, 77 percent of fetching bots not AI tools, SEO audit tools 21.7 percent (opens in a new tab) Checked 2026-10-26.
- SE Ranking, LLMs.txt and AI citations across nearly 300,000 domains, 7 November 2025: no effect on citation frequency (opens in a new tab) Checked 2026-10-26.
- Google Search Central, optimizing for generative AI features: Google Search ignores these AI text files (opens in a new tab) Checked 2026-10-26.
- Chrome for Developers, the Lighthouse agentic browsing audit: a server error is flagged, a missing file (404) is marked not applicable. Last updated 5 May 2026 (opens in a new tab) Checked 2026-10-26.
- OpenAI crawler documentation (no consumer of the file documented) (opens in a new tab) Checked 2026-10-26.
- Anthropic crawler documentation (no consumer of the file documented) (opens in a new tab) Checked 2026-10-26.
- Perplexity crawler documentation (no consumer of the file documented) (opens in a new tab) Checked 2026-10-26.
- Google Search Central: ecommerce site structure (opens in a new tab) Checked 2026-10-26.
CoversThis article covers the AI Search Essentials topic, the Website Audit feature and the llms.txt Generator tool.
Terms usedllms.txt, AI Crawler, Robots.txt, GPTBot and XML Sitemap.
Read this page asMarkdown: /blog/llms-txt.md.
All articlesEverything RankX AI publishes is listed on the blog index.
Ask an assistantAsk ChatGPT (opens in a new tab), Ask Claude (opens in a new tab) or Ask Perplexity (opens in a new tab).
Preferred sourceIf Google is your front door, you can add RankX AI as a preferred source (opens in a new tab), which asks your own results to surface more of what we publish.

