llms.txt a year on: what the data says
Adoption is up almost ninefold, AI crawlers barely ask for it, and Google has spoken. What a year of data says, and what we would advise today.
Our first article, last September, was about llms.txt, the plain text file meant to give AI tools a tidy summary of a website. Our advice then was that it was worth adding for software products with documentation, harmless for everyone else, and not something to pay for as "AI visibility".
There is much more evidence now than there was then.
Adoption grew fast
Originality.ai, which tracks more than three million websites, counted about 4,000 sites with an llms.txt file in June 2025. By May this year it counted just over 36,000, almost nine times as many. Files containing a site's full text, called llms-full.txt, grew from a couple of dozen to around 2,500.
That is real growth, helped by website platforms and SEO plugins that now create the file automatically. It is still a small share of the web.
The bots still are not reading it
The more important question is whether AI systems use the file. Two log studies this summer give a clear answer.
Ahrefs looked at May's server logs across 137,000 domains. Ninety-seven percent of llms.txt files received no requests at all. Of the requests that did arrive, the largest group, about a fifth, came from SEO audit tools checking whether the file exists. AI retrieval bots, the ones that fetch pages to answer a user's question, made up about 1% of requests.
A smaller study published in July by EZY.ai, a company that sells llms.txt tools, followed 83 sites for twelve weeks. OpenAI's GPTBot fetched robots.txt 3,990 times and llms.txt 7 times. Anthropic's ClaudeBot: 3,120 against 9. PerplexityBot never asked for llms.txt at all. Meta's crawler was the one exception, fetching it slightly more often than robots.txt.
None of the major AI companies has said that its crawlers use the file.
Google said it plainly
In June, Google added a section on llms.txt to its guidance on optimising for AI features in Search. It says Google Search does not use these files, that it is completely fine to keep one, and that doing so "will neither harm nor help your site's visibility or rankings in Google Search". Its documentation also says you do not need special AI files, markup or Markdown versions of pages to appear in AI features.
There is one mixed signal. In May, Google's Lighthouse testing tool added an experimental "agentic browsing" section that checks for an llms.txt file, along with WebMCP and the page's accessibility. That is about AI agents using a site, not about search rankings, and a missing file is marked "not applicable" rather than a failure.
The spec changed direction
Earlier this month, the llms.txt proposal published a second version. The interesting change is a shift in emphasis from one summary file to individual pages. Each page can now point to a clean Markdown version of itself and to the llms.txt file that covers it, using standard link tags. Separate llms.txt files can cover different sections of a large site.
This matches how agents actually behave. A coding assistant or research agent sent to one page wants that page's content without the menus and scripts, much more than it wants a site-wide summary.
What we would advise today
Our view from last year mostly holds, with one adjustment.
- If you publish documentation, an API or a developer product, keep or add llms.txt, and consider offering Markdown versions of your key pages with the new link tags. Coding assistants do fetch these when a user points them at your docs.
- For a typical business website, the file is still optional. If your platform creates it automatically, leave it. If not, it is not worth a line in your budget.
- Do not pay anyone to "optimise your llms.txt" for AI answers or rankings. The evidence says it does nothing for either.
What does matter is much the same as last year, plus one addition. Make sure your pages show their content without needing JavaScript. Decide which AI crawlers to allow; since July, Cloudflare lets every customer control search, agent and training bots separately. Put the facts customers ask about in plain text on your pages. And keep your Google Business Profile and reviews in order, because AI answers about local businesses often draw on them.
New formats for AI appear every few months. The ones that end up mattering are the ones the big AI systems actually read, and the cheapest way to find out is to wait for the logs.
