Robots.txt for AI Crawlers: Which Bots to Allow or Block
Robots.txt for AI crawlers works by naming each bot separately, because vendors split training from search. Blocking GPTBot, ClaudeBot or Google-Extended opts you out of model training without removing you from ChatGPT search, Claude or Google Search. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those answers. Some user-triggered fetchers may ignore the file.
Key points
- Training bots and search bots are separate switches, so blocking GPTBot does not remove you from ChatGPT search and blocking OAI-SearchBot does.
- Google-Extended has no effect on Google Search, AI Overviews or AI Mode, which follow your Googlebot rules.
- OpenAI, Perplexity and Google say their user-triggered fetchers may or generally do ignore robots.txt, while Anthropic says its bots honour it.
- Microsoft has no separate Copilot crawler and will honour a new robots.txt training token from 1 February 2027.
- Judge the result from verified crawler hits in your logs and from repeated AI answers reported with a sample size, never from a single answer.
What is robots.txt for AI crawlers?
Robots.txt for AI crawlers is the same plain text file search engines have always read, with extra groups naming AI bots. It sits at the root of each host and tells a named user agent which URLs it may fetch.
Two limits matter before you edit anything. First, the file is a request. Google's robots.txt introduction says the file cannot enforce behaviour and that obeying it is up to the crawler. Second, it controls crawling, not what a model already knows or what other sites say about you.
It is also a different thing from llms.txt, which is a proposed content map and not an access control. We cover that file in does llms.txt work.
Which AI crawlers exist, and do they obey robots.txt?
Each large AI vendor now documents at least two agents: one for model training and one for search, often with a third for fetches a user triggers. The table below uses only each vendor's own documentation, read on 11 November 2026.
| User agent or token | Owner | What it is for | Type | robots.txt, in the vendor's words |
|---|---|---|---|---|
GPTBot | OpenAI | Collects content that may be used to train OpenAI's models | Training | Disallowing it signals the content should not be used for training |
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | Search | Controlled by robots.txt. Opted-out sites are left out of search answers but can still show as navigational links |
ChatGPT-User | OpenAI | Visits a page when a ChatGPT or Custom GPT user's request needs it | User-triggered | Rules "may not apply" because a user started the action |
ClaudeBot | Anthropic | Collects content that could help train Anthropic's models | Training | Honoured. Blocking signals that future material should stay out of training sets |
Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results | Search | Honoured. Blocking may reduce your visibility in search results |
Claude-User | Anthropic | Fetches pages when a Claude user asks a question | User-triggered | Honoured. Blocking stops retrieval of your content for user queries |
PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results. Not used to crawl for foundation models | Search | Controlled by robots.txt. Perplexity recommends allowing it |
Perplexity-User | Perplexity | Visits a page to help answer a user's question. Not used for training | User-triggered | "Generally ignores" robots.txt because a user requested the fetch |
Googlebot | Crawls for Google Search and all Search features, which include AI Overviews and AI Mode | Search | Obeyed. This is the control for how Search crawls your site | |
Google-Extended | A token, not a crawler. Governs Gemini model training and grounding in Gemini Apps and Vertex AI | Training and grounding control | Obeyed as a robots.txt token. No effect on Search inclusion or ranking | |
Google-Agent, Google-GeminiNotebook | Agents browsing for a user, and source URLs added to Gemini Notebook | User-triggered | These fetchers "generally ignore robots.txt rules" | |
Applebot | Apple | Powers search in Spotlight, Siri and Safari. Data may also help train Apple's models | Search, plus training unless opted out | Respected. Falls back to your Googlebot rules if none name Applebot |
Applebot-Extended | Apple | A token, not a crawler. Opts already-crawled content out of generative AI training | Training control | Respected. Pages can still appear in Apple search results |
CCBot | Common Crawl | Builds a free public copy of the web that anyone can download | Open dataset | Checks robots.txt before fetching and obeys Crawl-delay |
Bingbot and __usage_ai-training | Microsoft | Bingbot crawls for Bing. The new token sets a training preference for a site or path | Search, plus training control | Bingbot rules still decide crawling. The training token is honoured from 1 February 2027 |
Sources: OpenAI, Anthropic, Perplexity, Google's common crawlers and user-triggered fetchers pages, Apple, Common Crawl and Microsoft Bing.
Read the last column as each vendor's claim about itself. Nobody outside these companies can audit compliance across the whole web.
The user-triggered rows deserve a second look. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity and Google both say their user-requested fetchers generally ignore the file. Anthropic is the exception: its article says its bots honour robots.txt, and that blocking Claude-User stops retrieval for user queries. If a page must never be fetched, robots.txt is the wrong tool. Put it behind a login or block it at the server.
Does blocking GPTBot remove you from ChatGPT search?
No. Blocking a training bot does not remove you from that vendor's search product, and blocking a search bot does not opt you out of training. The vendors treat these as separate switches.
On OAI-SearchBot vs GPTBot, OpenAI's crawler page says each setting is independent of the others. Its own example is a site that allows OAI-SearchBot so it can appear in search while disallowing GPTBot. The reverse is the costly mistake: block OAI-SearchBot and OpenAI says your site is left out of ChatGPT search answers.
The same split applies elsewhere:
- Anthropic.
ClaudeBotis the training crawler.Claude-SearchBotandClaude-Useraffect whether Claude can find and retrieve your pages. - Google.
Google-Extendeddoes not affect Google Search. Google's documentation says it has no impact on inclusion in Search and is not a ranking signal. AI Overviews and AI Mode are Search features, and Google's AI features page names Googlebot rules as the control for those. Google-Extended covers Gemini training plus grounding in Gemini Apps and Vertex AI. - Apple.
Applebot-Extendednever crawls. Disallowing it opts content out of training while pages stay eligible for Spotlight, Siri and Safari results. - Microsoft. There is no separate Copilot crawler in Microsoft's documentation. Bingbot crawls, and new preferences set how content is used.
Microsoft's change is recent. A Bing post from October 2026 introduces a robots.txt token, __usage_ai-training, for training, and a page-level ai-grounding preference set by meta tag or HTTP header. Microsoft says neither changes eligibility for ordinary Bing results. It will honour them from 1 February 2027, replacing the older NOARCHIVE and NOCACHE tags. Google's crawler page says it intends to support the same training token in early 2027.
Should I block GPTBot and other training bots?
Block training bots if you object to your content being used to train models; allow them if you do not mind. For most brands it is a policy and licensing choice, because the vendors say it does not change search eligibility.
Here is what is known and what is not:
- Known. The vendors document training and search as independent controls, so opting out of training should not cost you citations in their search products.
- Not known. No vendor documents whether being in training data changes how often a model names your brand when it answers without searching. Nobody outside the vendors can measure that directly.
- Limited reach. Anthropic describes a ClaudeBot block as a signal about future material. None of the pages we read promise removal of content already collected.
Common Crawl is a special case. CCBot is not an AI company's bot. It feeds an open archive that Common Crawl's FAQ says is free for researchers, companies and individuals. Blocking it keeps new pages out of that archive, whoever uses it.
If your goal is to be named more often, robots.txt only removes barriers. The work that earns mentions is covered in how to get your brand mentioned in ChatGPT.
What should my robots.txt say?
Pick one of two starting points: stay citable in AI search while opting out of training, or allow everything. Both files below are complete. Replace the sitemap URL with your own.
Option 1: be citable in AI search but opt out of training
# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: __usage_ai-training
Disallow: /
# Optional: also stops grounding in Gemini Apps
User-agent: Google-Extended
Disallow: /
# Stay citable in AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Three notes on this file. The Google-Extended group is optional because it has a cost: Google lists grounding in Gemini Apps among the things it limits. Leave it out if appearing in the Gemini app matters to you, and read how to show up in Google AI Mode and Gemini first.
The __usage_ai-training line does nothing at Microsoft until 1 February 2027, but it is safe to add now. Apple says Applebot data may be used for training unless you disallow Applebot-Extended, which is why that token sits in the first group.
Option 2: allow everything
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
This permits every crawler that reads the file, for training and search alike.
How do I check what my site currently serves?
Fetch the live file yourself, then confirm from logs that the bots you allowed are arriving.
- Open the file. Visit
https://yourdomain/robots.txtin a browser. Repeat for every subdomain; Anthropic's article notes that rules are needed per subdomain. - Search it for the names in the table. Look for a
Disallow: /under any search bot, and underUser-agent: *. - Test the server, not only the file. Run
curl -I -A "OAI-SearchBot" https://yourdomain/and check for a 200 response. A firewall rule can refuse a bot that robots.txt allows. - Check Google's view. The robots.txt report in Search Console shows the files Google found, the last crawl time and any parsing issues. You can request a recrawl there.
- Read your logs. Filter by user agent, then confirm the IPs. OpenAI, Anthropic, Perplexity and Common Crawl publish their IP ranges as JSON. Google warns that user agent strings can be spoofed.
- Wait before judging. OpenAI says search can take about 24 hours to reflect a robots.txt change. Microsoft says its preferences apply only after a recrawl.
Avoid blocking by IP address as a substitute. Anthropic warns that it may not work reliably, because the bot can then no longer read your robots.txt at all.
How to measure whether it worked
Measure two things: whether the right bots now crawl you, and whether AI answers name or cite you more often. The first shows up in logs within days. The second needs repeated runs, because one answer proves nothing.
For crawling, count requests per verified user agent for the weeks before and after the change. A search bot that was blocked should start appearing. A training bot you disallowed should stop requesting pages other than robots.txt.
For visibility, fix a set of questions your customers ask and run the same set on a schedule. AI answers vary from run to run, so report a share with its sample size. If 9 of 30 answers name you, that is 30%, but the 95% interval runs from about 17% to 48%. At 90 of 300 it narrows to about 25% to 35%. A move from 30% to 37% on 30 answers is noise.
Change one thing at a time and give it several weeks. Google says recrawling can take from several days to several months.
You can run the prompts by hand in a spreadsheet. CiteSieve automates it: it asks six AI engines your customers' questions once a day and shows the share of answers naming your brand with a 95% interval. Its site audit also checks robots.txt. For the method behind the numbers, see AI share of voice.
Questions and answers
What is the difference between OAI-SearchBot and GPTBot?
GPTBot collects content that OpenAI may use to train its models. OAI-SearchBot finds pages so ChatGPT's search features can surface them. OpenAI says the two settings are independent, so you can disallow GPTBot and still appear in ChatGPT search. If you allow both, OpenAI says it may use one crawl for both purposes.
Does Google-Extended affect AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode are Search features, controlled by Googlebot rules and by snippet controls such as nosnippet and noindex. Google-Extended limits Gemini model training and grounding in Gemini Apps and Vertex AI.
How long does a robots.txt change take to work for AI bots?
It varies by vendor. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update. Google says recrawling can take from several days to several months. Microsoft says its new training and grounding preferences only take effect once a page has been recrawled.
Is blocking AI bots by IP address better than robots.txt?
Usually not as a replacement. Anthropic advises against it, saying an IP block may not work reliably because the bot can no longer read your robots.txt. IP lists are more useful for verification: OpenAI, Anthropic, Perplexity and Common Crawl publish their ranges, so you can confirm that a visitor claiming to be a bot is real.
Sources
- OpenAI: Overview of OpenAI crawlers
- Google Search Central: Google's common crawlers (Googlebot, Google-Extended)
- Google Search Central: Google user-triggered fetchers
- Google Search Central: AI features and your website
- Google Search Central: Introduction to robots.txt
- Google Search Console Help: robots.txt report
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Apple Support: About Applebot
- Common Crawl: CCBot
- Common Crawl: Frequently asked questions
- Microsoft Bing: Your content, your choice: clearer preferences for training and grounding
Sources were read on 11 November 2026. Engines and their documentation change; check the linked page for the current wording.