Reference

Robots.txt for AI Crawlers: Which Bots to Allow or Block

Robots.txt for AI crawlers works by naming each bot separately, because vendors split training from search. Blocking GPTBot, ClaudeBot or Google-Extended opts you out of model training without removing you from ChatGPT search, Claude or Google Search. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those answers. Some user-triggered fetchers may ignore the file.

Key points

  • Training bots and search bots are separate switches, so blocking GPTBot does not remove you from ChatGPT search and blocking OAI-SearchBot does.
  • Google-Extended has no effect on Google Search, AI Overviews or AI Mode, which follow your Googlebot rules.
  • OpenAI, Perplexity and Google say their user-triggered fetchers may or generally do ignore robots.txt, while Anthropic says its bots honour it.
  • Microsoft has no separate Copilot crawler and will honour a new robots.txt training token from 1 February 2027.
  • Judge the result from verified crawler hits in your logs and from repeated AI answers reported with a sample size, never from a single answer.

What is robots.txt for AI crawlers?

Robots.txt for AI crawlers is the same plain text file search engines have always read, with extra groups naming AI bots. It sits at the root of each host and tells a named user agent which URLs it may fetch.

Two limits matter before you edit anything. First, the file is a request. Google's robots.txt introduction says the file cannot enforce behaviour and that obeying it is up to the crawler. Second, it controls crawling, not what a model already knows or what other sites say about you.

It is also a different thing from llms.txt, which is a proposed content map and not an access control. We cover that file in does llms.txt work.

Which AI crawlers exist, and do they obey robots.txt?

Each large AI vendor now documents at least two agents: one for model training and one for search, often with a third for fetches a user triggers. The table below uses only each vendor's own documentation, read on 11 November 2026.

User agent or tokenOwnerWhat it is forTyperobots.txt, in the vendor's words
GPTBotOpenAICollects content that may be used to train OpenAI's modelsTrainingDisallowing it signals the content should not be used for training
OAI-SearchBotOpenAISurfaces sites in ChatGPT's search featuresSearchControlled by robots.txt. Opted-out sites are left out of search answers but can still show as navigational links
ChatGPT-UserOpenAIVisits a page when a ChatGPT or Custom GPT user's request needs itUser-triggeredRules "may not apply" because a user started the action
ClaudeBotAnthropicCollects content that could help train Anthropic's modelsTrainingHonoured. Blocking signals that future material should stay out of training sets
Claude-SearchBotAnthropicIndexes content to improve Claude's search resultsSearchHonoured. Blocking may reduce your visibility in search results
Claude-UserAnthropicFetches pages when a Claude user asks a questionUser-triggeredHonoured. Blocking stops retrieval of your content for user queries
PerplexityBotPerplexitySurfaces and links sites in Perplexity search results. Not used to crawl for foundation modelsSearchControlled by robots.txt. Perplexity recommends allowing it
Perplexity-UserPerplexityVisits a page to help answer a user's question. Not used for trainingUser-triggered"Generally ignores" robots.txt because a user requested the fetch
GooglebotGoogleCrawls for Google Search and all Search features, which include AI Overviews and AI ModeSearchObeyed. This is the control for how Search crawls your site
Google-ExtendedGoogleA token, not a crawler. Governs Gemini model training and grounding in Gemini Apps and Vertex AITraining and grounding controlObeyed as a robots.txt token. No effect on Search inclusion or ranking
Google-Agent, Google-GeminiNotebookGoogleAgents browsing for a user, and source URLs added to Gemini NotebookUser-triggeredThese fetchers "generally ignore robots.txt rules"
ApplebotApplePowers search in Spotlight, Siri and Safari. Data may also help train Apple's modelsSearch, plus training unless opted outRespected. Falls back to your Googlebot rules if none name Applebot
Applebot-ExtendedAppleA token, not a crawler. Opts already-crawled content out of generative AI trainingTraining controlRespected. Pages can still appear in Apple search results
CCBotCommon CrawlBuilds a free public copy of the web that anyone can downloadOpen datasetChecks robots.txt before fetching and obeys Crawl-delay
Bingbot and __usage_ai-trainingMicrosoftBingbot crawls for Bing. The new token sets a training preference for a site or pathSearch, plus training controlBingbot rules still decide crawling. The training token is honoured from 1 February 2027

Sources: OpenAI, Anthropic, Perplexity, Google's common crawlers and user-triggered fetchers pages, Apple, Common Crawl and Microsoft Bing.

Read the last column as each vendor's claim about itself. Nobody outside these companies can audit compliance across the whole web.

The user-triggered rows deserve a second look. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity and Google both say their user-requested fetchers generally ignore the file. Anthropic is the exception: its article says its bots honour robots.txt, and that blocking Claude-User stops retrieval for user queries. If a page must never be fetched, robots.txt is the wrong tool. Put it behind a login or block it at the server.

No. Blocking a training bot does not remove you from that vendor's search product, and blocking a search bot does not opt you out of training. The vendors treat these as separate switches.

On OAI-SearchBot vs GPTBot, OpenAI's crawler page says each setting is independent of the others. Its own example is a site that allows OAI-SearchBot so it can appear in search while disallowing GPTBot. The reverse is the costly mistake: block OAI-SearchBot and OpenAI says your site is left out of ChatGPT search answers.

The same split applies elsewhere:

  • Anthropic. ClaudeBot is the training crawler. Claude-SearchBot and Claude-User affect whether Claude can find and retrieve your pages.
  • Google. Google-Extended does not affect Google Search. Google's documentation says it has no impact on inclusion in Search and is not a ranking signal. AI Overviews and AI Mode are Search features, and Google's AI features page names Googlebot rules as the control for those. Google-Extended covers Gemini training plus grounding in Gemini Apps and Vertex AI.
  • Apple. Applebot-Extended never crawls. Disallowing it opts content out of training while pages stay eligible for Spotlight, Siri and Safari results.
  • Microsoft. There is no separate Copilot crawler in Microsoft's documentation. Bingbot crawls, and new preferences set how content is used.

Microsoft's change is recent. A Bing post from October 2026 introduces a robots.txt token, __usage_ai-training, for training, and a page-level ai-grounding preference set by meta tag or HTTP header. Microsoft says neither changes eligibility for ordinary Bing results. It will honour them from 1 February 2027, replacing the older NOARCHIVE and NOCACHE tags. Google's crawler page says it intends to support the same training token in early 2027.

Should I block GPTBot and other training bots?

Block training bots if you object to your content being used to train models; allow them if you do not mind. For most brands it is a policy and licensing choice, because the vendors say it does not change search eligibility.

Here is what is known and what is not:

  • Known. The vendors document training and search as independent controls, so opting out of training should not cost you citations in their search products.
  • Not known. No vendor documents whether being in training data changes how often a model names your brand when it answers without searching. Nobody outside the vendors can measure that directly.
  • Limited reach. Anthropic describes a ClaudeBot block as a signal about future material. None of the pages we read promise removal of content already collected.

Common Crawl is a special case. CCBot is not an AI company's bot. It feeds an open archive that Common Crawl's FAQ says is free for researchers, companies and individuals. Blocking it keeps new pages out of that archive, whoever uses it.

If your goal is to be named more often, robots.txt only removes barriers. The work that earns mentions is covered in how to get your brand mentioned in ChatGPT.

What should my robots.txt say?

Pick one of two starting points: stay citable in AI search while opting out of training, or allow everything. Both files below are complete. Replace the sitemap URL with your own.

Option 1: be citable in AI search but opt out of training

# Opt out of AI training User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: CCBot User-agent: __usage_ai-training Disallow: / # Optional: also stops grounding in Gemini Apps User-agent: Google-Extended Disallow: / # Stay citable in AI search User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Googlebot User-agent: Bingbot User-agent: Applebot Allow: / User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml

Three notes on this file. The Google-Extended group is optional because it has a cost: Google lists grounding in Gemini Apps among the things it limits. Leave it out if appearing in the Gemini app matters to you, and read how to show up in Google AI Mode and Gemini first.

The __usage_ai-training line does nothing at Microsoft until 1 February 2027, but it is safe to add now. Apple says Applebot data may be used for training unless you disallow Applebot-Extended, which is why that token sits in the first group.

Option 2: allow everything

User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml

This permits every crawler that reads the file, for training and search alike.

How do I check what my site currently serves?

Fetch the live file yourself, then confirm from logs that the bots you allowed are arriving.

  1. Open the file. Visit https://yourdomain/robots.txt in a browser. Repeat for every subdomain; Anthropic's article notes that rules are needed per subdomain.
  2. Search it for the names in the table. Look for a Disallow: / under any search bot, and under User-agent: *.
  3. Test the server, not only the file. Run curl -I -A "OAI-SearchBot" https://yourdomain/ and check for a 200 response. A firewall rule can refuse a bot that robots.txt allows.
  4. Check Google's view. The robots.txt report in Search Console shows the files Google found, the last crawl time and any parsing issues. You can request a recrawl there.
  5. Read your logs. Filter by user agent, then confirm the IPs. OpenAI, Anthropic, Perplexity and Common Crawl publish their IP ranges as JSON. Google warns that user agent strings can be spoofed.
  6. Wait before judging. OpenAI says search can take about 24 hours to reflect a robots.txt change. Microsoft says its preferences apply only after a recrawl.

Avoid blocking by IP address as a substitute. Anthropic warns that it may not work reliably, because the bot can then no longer read your robots.txt at all.

How to measure whether it worked

Measure two things: whether the right bots now crawl you, and whether AI answers name or cite you more often. The first shows up in logs within days. The second needs repeated runs, because one answer proves nothing.

For crawling, count requests per verified user agent for the weeks before and after the change. A search bot that was blocked should start appearing. A training bot you disallowed should stop requesting pages other than robots.txt.

For visibility, fix a set of questions your customers ask and run the same set on a schedule. AI answers vary from run to run, so report a share with its sample size. If 9 of 30 answers name you, that is 30%, but the 95% interval runs from about 17% to 48%. At 90 of 300 it narrows to about 25% to 35%. A move from 30% to 37% on 30 answers is noise.

Change one thing at a time and give it several weeks. Google says recrawling can take from several days to several months.

You can run the prompts by hand in a spreadsheet. CiteSieve automates it: it asks six AI engines your customers' questions once a day and shows the share of answers naming your brand with a 95% interval. Its site audit also checks robots.txt. For the method behind the numbers, see AI share of voice.

Questions and answers

What is the difference between OAI-SearchBot and GPTBot?

GPTBot collects content that OpenAI may use to train its models. OAI-SearchBot finds pages so ChatGPT's search features can surface them. OpenAI says the two settings are independent, so you can disallow GPTBot and still appear in ChatGPT search. If you allow both, OpenAI says it may use one crawl for both purposes.

Does Google-Extended affect AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are Search features, controlled by Googlebot rules and by snippet controls such as nosnippet and noindex. Google-Extended limits Gemini model training and grounding in Gemini Apps and Vertex AI.

How long does a robots.txt change take to work for AI bots?

It varies by vendor. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update. Google says recrawling can take from several days to several months. Microsoft says its new training and grounding preferences only take effect once a page has been recrawled.

Is blocking AI bots by IP address better than robots.txt?

Usually not as a replacement. Anthropic advises against it, saying an IP block may not work reliably because the bot can no longer read your robots.txt. IP lists are more useful for verification: OpenAI, Anthropic, Perplexity and Common Crawl publish their ranges, so you can confirm that a visitor claiming to be a bot is real.

Sources

  1. OpenAI: Overview of OpenAI crawlers
  2. Google Search Central: Google's common crawlers (Googlebot, Google-Extended)
  3. Google Search Central: Google user-triggered fetchers
  4. Google Search Central: AI features and your website
  5. Google Search Central: Introduction to robots.txt
  6. Google Search Console Help: robots.txt report
  7. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  8. Perplexity: Perplexity crawlers
  9. Apple Support: About Applebot
  10. Common Crawl: CCBot
  11. Common Crawl: Frequently asked questions
  12. Microsoft Bing: Your content, your choice: clearer preferences for training and grounding

Sources were read on 11 November 2026. Engines and their documentation change; check the linked page for the current wording.

More guides

Find out what AI says about your brand.

A site audit on the page in seconds, then live answers by email. No account, no card.