Explainer

Does llms.txt Work? What It Is and What the Evidence Shows

llms.txt is a proposed Markdown file at /llms.txt that gives AI agents a curated map of a site. It does not work as a visibility lever today: Google Search ignores it, no major AI vendor documents using it for answers, and published log studies show most files are never fetched. It can still help documentation sites and coding agents.

Key points

  • llms.txt is a community proposal from September 2024, not a standard any search engine or AI vendor has committed to.
  • Google's AI optimisation guide says Google Search ignores the file, so it neither helps nor harms rankings or AI Overviews.
  • Ahrefs found in June 2026 that 97% of valid llms.txt files across 137,210 domains received no requests in a month.
  • The file is reasonable for documentation and agent-facing sites, where coding agents do fetch it, and the cost is low.
  • Judge any change by repeated answer runs with enough samples, not by a single prompt before and after.

What is llms.txt?

llms.txt is a proposed Markdown file, placed at /llms.txt, that gives AI models and agents a short, curated map of a site. It is a community proposal, not a standard that search engines or AI companies have agreed to follow.

Jeremy Howard published the proposal at llmstxt.org on 3 September 2024. The page now describes itself as version 2 and was last modified on 10 August 2026. Its argument is practical: web pages are built for people, and their content sits inside menus, adverts and scripts. A model also cannot fit most whole sites into its context window.

The proposal has a second half that gets less attention. It asks sites to offer a clean Markdown copy of useful pages at the same address with .md added. The llms.txt file then links to those copies.

Two points from the proposal matter for marketers. First, its authors expected the file to be used mainly at inference time, when an agent fetches it on demand, not for training. Second, it names coding agents reading library documentation and chat assistants with search as the intended readers. Nothing in it promises better rankings or more citations.

The file is also not a permissions tool. The proposal says it complements robots.txt, which states what automated access is acceptable, and does not replace a sitemap. If your question is about blocking or allowing bots, read our guide to robots.txt for AI crawlers instead.

What does a valid llms.txt file look like?

A valid file is plain Markdown with one required element: an H1 holding the site or project name. Everything else is optional, but the order is fixed.

After the H1 comes a blockquote with a short summary. Then come any free-text notes, as long as they are not headings. Then come H2 sections, each holding a list of links with an optional note after a colon. A section titled Optional holds secondary links that an agent may skip when it needs a shorter context.

A short example that follows those rules, for an invented shop:

  • # Example Tea Co
  • > Loose-leaf tea sold online in the UK, retail and wholesale.
  • ## Products
  • - [Tea range](https://example.com/teas.md): every tea, with pack sizes
  • - [Wholesale](https://example.com/wholesale.md): trade terms and minimum order
  • ## Optional
  • - [Our story](https://example.com/about.md)

The file can sit at the site root or under a subpath such as /docs/llms.txt. Under the proposal, a file covers the URLs beneath its own path.

What does Google say about llms.txt?

Google Search ignores llms.txt. Google's AI optimisation guide, last updated on 10 July 2026, says you do not need new machine-readable files, AI text files or Markdown to appear in Search.

The same guide is careful about the other direction. It says publishing the file is fine if you want it for other services, and that it will neither harm nor help visibility or rankings in Google Search. That covers AI Overviews and AI Mode, because the guide treats generative AI search as part of Search.

Google staff said much the same before the guide existed. Search Engine Journal reported on 17 April 2025 that John Mueller, replying in a Reddit thread, said no AI service had announced it was using the file. He compared it to the keywords meta tag. A later Search Engine Journal piece, dated 20 May 2026, adds that Gary Illyes confirmed Google was not pursuing llms.txt.

There is one wrinkle. Chrome's Lighthouse tool now includes an llms.txt check in its Agentic Browsing category. The Lighthouse documentation, updated on 5 May 2026, calls the file an emerging convention for agents. A missing file is marked not applicable, not failed. Search Engine Journal read the gap as two different questions: visibility in Search, and readiness for browser agents. Google has not commented on it, according to that article.

Do ChatGPT, Claude and Perplexity use llms.txt?

None of the three documents llms.txt as something that affects which sites its answers cite. Their public crawler pages describe robots.txt and IP ranges as the controls, and nothing else.

OpenAI's crawler page lists OAI-SearchBot for ChatGPT search, GPTBot for training and ChatGPT-User for visits a user triggers. It says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic's help article lists ClaudeBot, Claude-User and Claude-SearchBot, and does not mention llms.txt at all. Perplexity's page lists PerplexityBot and Perplexity-User, and recommends allowing PerplexityBot if you want to appear in its results.

What these vendors do is a little different from what they say. The OpenAI and Perplexity pages each point readers to an llms.txt index of their own documentation. The proposal's site also reports that AI labs publish the file for their developer docs. So the labs use llms.txt as publishers, for developers and coding agents. That is not evidence that their search products read yours.

Nobody outside these companies knows whether any retrieval system quietly gives the file weight. The honest position is that no vendor has said so, and the log data below gives little sign of it.

What do the studies and server logs show?

The published data points one way: most llms.txt files are never requested, and having one shows no measurable link to being cited. Each study has limits, and all three come from companies that sell SEO or AI visibility tools.

Source and dateWhat was measuredFindingMain limit
Ahrefs, 15 June 2026Requests to /llms.txt across 137,210 domains in May 202628% had a valid file; 97% of those files got zero requests that monthSample skews technical; a fetch does not prove the file was used
OtterlyAI, 5 February 2026AI bot requests in one test site's server logs over 90 days84 of more than 62,100 AI bot requests went to /llms.txt, about 0.1%One site, no control, bots not named
SE Ranking, article dated 15 October 2026llms.txt presence against citation counts for nearly 300,000 domains10.13% had a file; no correlation with how often a domain was citedAssociation only; models and time window not stated

The Ahrefs breakdown is the most useful part. Of roughly 22,000 requests to the files that did get traffic, SEO audit tools made 21.7%. All AI bots combined made 19.5%, and AI retrieval bots, the kind that fetch pages for answers, made 1.1%. Ahrefs also found that Claude-Code, a coding agent, was the second-largest named AI fetcher.

That pattern fits the proposal's own description of who the file is for. Coding agents fetch it. Search and answer systems, on this evidence, mostly do not.

What remains disputed is the future. Supporters argue that agent use is growing and the file costs little. Sceptics point to Mueller's keywords meta tag comparison. Neither side has data showing a visibility gain today.

When is llms.txt still worth publishing?

Publish one when software agents, not search engines, are a real audience for your content. For a typical brand or shop site, the evidence does not support treating it as a visibility task.

Reasonable cases include:

  • Documentation and API sites. Developers point coding agents at docs, and the Ahrefs data shows a coding agent among the main fetchers.
  • Products that agents must operate. Lighthouse's agent audit describes the file as a way to save an agent from crawling the site to learn its structure.
  • Sites where it is generated for you. The proposal notes that documentation platforms now produce the file automatically.

The cost is low but not zero. A hand-written file goes stale when pages move, and a stale map is worse than none. If you publish one, generate it from the same source as your sitemap. Keep it honest too. It should describe the pages people see, not a keyword-heavy version written for machines.

For agencies, the advice to clients is simple. It is harmless to add and fair to charge little for. It should not appear on a report as a reason AI visibility went up.

What should you do instead for AI visibility?

Spend the time on access, content and reputation, which the vendors do document. In order:

  1. Check crawler access. Make sure robots.txt and your firewall allow OAI-SearchBot, PerplexityBot and Claude-SearchBot, plus Googlebot. Each vendor's own page ties its search bot to whether you appear in its results.
  2. Meet Google's technical basics. Google's guide asks for crawlable pages, sound JavaScript handling and a good page experience. It says no special schema or content chunking is required.
  3. Publish content that is not a commodity. The guide's main recommendation is unique, helpful content written for people. Original data and first-hand experience are harder for a model to get elsewhere.
  4. Keep product and business data current. Google's guide points to Merchant Center and Google Business Profile for product and local visibility.
  5. Earn mentions on other sites. Our guide on how to get your brand mentioned in ChatGPT covers this step.

Google-specific steps are covered in showing up in Google AI Mode and Gemini.

How to measure whether it worked

Measure two separate things: whether anything fetches the file, and whether your brand appears in more answers. The first is a log question. The second needs repeated runs, because AI answers vary from one run to the next.

Fetches. Filter your server logs for requests to /llms.txt and group them by user agent. Compare the count with requests to an ordinary content page over the same period, as OtterlyAI did. Separate audit tools from AI retrieval bots, as Ahrefs did. A handful of hits from SEO checkers tells you nothing about answers.

Answers. Write a fixed list of the questions your customers ask. Run each one many times, on each assistant, before you publish the file and again afterwards. Record whether your brand is named and which sources are cited. A single run is an anecdote.

Sample size decides what you can claim. If your brand appears in 12 of 40 answers, the share is 30%, but the 95% interval runs from about 18% to 45%. With 120 of 400 answers it narrows to about 26% to 35%. A move from 30% to 34% on 40 answers is noise.

Change one thing at a time where you can. If the file goes live in the same week as new content and a digital PR push, you cannot credit the file. For Google's surfaces, the guide also points to the generative AI performance report in Search Console.

Tools can do the repetition for you. CiteSieve, an AI search visibility tracker from Braivex, asks ChatGPT, Google AI Mode, Gemini, Perplexity, Copilot and Google AI Overviews your customers' questions once a day. It reports the share of answers naming your brand, with the number of answers and a 95% interval. Our guide to AI share of voice explains how to read those figures.

Questions and answers

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may access, and OpenAI, Anthropic and Perplexity all document it as the control for their bots. llms.txt grants or blocks nothing. It is a Markdown list of pages a site suggests an agent should read. The proposal itself says the file complements robots.txt and does not replace a sitemap.

Will adding llms.txt hurt my SEO?

Not in Google Search. Google's AI optimisation guide says the file will neither harm nor help visibility or rankings, because Search ignores it. The practical risk is maintenance: a file that lists moved or deleted pages misleads any agent that does read it, so generate it automatically if you publish one.

What is the difference between llms.txt and the .md pages in the proposal?

llms.txt is the index file. The proposal also asks sites to publish clean Markdown copies of useful pages at the same address with .md added, and the index links to those copies. Google's guide says you do not need Markdown versions to appear in Search, so treat both parts as optional extras for agents.

Why does Lighthouse check for llms.txt if Google Search ignores it?

They answer different questions. Lighthouse's Agentic Browsing category looks at whether browser agents can use a site, and its documentation calls llms.txt an emerging convention. A missing file is marked not applicable, not failed. Google's Search guidance is about ranking and AI features in Search, where the file plays no part.

Sources

  1. The /llms.txt file (llmstxt.org proposal)
  2. Google Search Central: AI optimisation guide
  3. Chrome for Developers: Lighthouse llms.txt audit
  4. OpenAI: Overview of OpenAI crawlers
  5. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  6. Perplexity: Perplexity crawlers
  7. Ahrefs: We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (15 June 2026)
  8. OtterlyAI: Llms.txt Experiment (5 February 2026)
  9. SE Ranking: Does LLMs.txt impact your AI visibility and citations?
  10. Search Engine Journal: Google says llms.txt comparable to keywords meta tag (17 April 2025)
  11. Search Engine Journal: Google's llms.txt guidance depends on which product you ask (20 May 2026)

Sources were read on 11 November 2026. Engines and their documentation change; check the linked page for the current wording.

More guides

Find out what AI says about your brand.

A site audit on the page in seconds, then live answers by email. No account, no card.