Skip to content

AI Crawlers, Search Access and Training Controls

See how AI search crawlers, training bots and user-triggered fetches differ, with current controls for Google, OpenAI, Anthropic and Perplexity.

Separate switches beside a server illustrating independent crawler controls
Jack Lee
6 Min Read
Share
On This Page

AI crawlers are automated programs that visit websites for AI products. Some collect material for model training; others help a search service discover pages. The access you allow can depend on that purpose.

You can allow search while declining training on platforms that provide separate controls.

Three Different Jobs Behind an AI Visit

  • Search discovery: an automated crawler finds pages that a product may retrieve or link to when answering a question.
  • Model training: collected material may help develop a future model. This is a different use from looking up a current webpage.
  • User-triggered retrieval: someone asks an AI product a question or supplies a URL, and the product fetches a page to respond. This is an on-demand request rather than a scheduled crawl.

These jobs can involve the same page but different controls. Allowing a visit does not guarantee that the page will be cited. Our AI search introduction explains how retrieved sources fit into an answer.

Which Controls Belong to Which Platform?

robots.txt is a public text file containing crawler instructions. A user agent identifies the program making a request; a User-agent line in the file names the bot a group of rules applies to. Here are the providers' documented roles, checked in October 2026:

PlatformSearch and user requestsTraining or other use
OpenAIOAI-SearchBot: ChatGPT search.
ChatGPT-User: certain user-triggered visits.
GPTBot: collection of content that may be used to train general-purpose AI models.
GoogleGooglebot: Google Search crawling. Search Console separately controls participation in covered generative Search features.Google-Extended: a robots token covering future Gemini training and specified Gemini grounding uses.
AnthropicClaude-SearchBot: search discovery.
Claude-User: retrieval for user questions.
ClaudeBot: collection for potential model training.
PerplexityPerplexityBot: search discovery.
Perplexity-User: user-requested retrieval.
Perplexity says these agents are not used to collect content for training general-purpose AI models.

User-triggered visits need separate attention. OpenAI says robots rules may not apply to ChatGPT-User; Perplexity says its user fetcher generally ignores them. Anthropic says its bots, including Claude-User, honor those rules. Recheck the provider's documentation before changing a policy.

Can You Allow Search While Declining Training?

Yes, where the provider separates those choices. Consider a kitchen-appliance website that wants its installation guides accessible to ChatGPT search, but declines OpenAI's training crawl. It also wants crawlers to skip its cart pages.

In this example, the search bot can fetch /guides/oven-installation/ but must skip /cart/; GPTBot is disallowed throughout. Merge the relevant rules into your existing file, preserving other groups and sitemap entries:

User-agent: *
Disallow: /cart/

User-agent: OAI-SearchBot
Allow: /
Disallow: /cart/

User-agent: GPTBot
Disallow: /

Here, / means every path from the site root. Allow: / permits crawling, while Disallow: / asks the named bot not to crawl any page.

The repeated cart rule matters. A bot with a matching specific group uses that group; it does not automatically inherit the User-agent: * group. Within the search bot's group, the more specific /cart/ restriction takes precedence over Allow: /. Preserve your own exclusions when adapting this example.

Google's controls have a different scope. Google-Extended is a policy token, not a separate request name to look for in logs. It covers both specified training and grounding uses—supplying retrieved information to an AI answer—and does not affect ordinary Google Search inclusion or ranking.

For AI Overviews, AI Mode and covered generative Discover features, open Search Console → Settings → Search generative AI. Check whether the property includes, excludes or inherits its choice from a parent. This is a separate participation control, not a training setting. Our AI Overviews and AI Mode guide covers that distinction.

What Can Robots.txt Do—and What Can It Not Do?

robots.txt gives instructions to cooperating crawlers. It cannot enforce access protection, undo past training or guarantee removal of every reference to a URL. Keep private content behind authentication.

The file belongs at the site's root, such as https://example.com/robots.txt. Its scope is that protocol, host and port; a subdomain needs its own file. Check each site version you serve rather than assuming one file covers everything.

A permitted bot can still be blocked by a web application firewall (WAF), a CAPTCHA or a server error. A robots check and a successful page request therefore answer different questions. Our robots.txt guide covers rule syntax; an optional llms.txt file does not replace these controls.

A Practical Access Audit for One Site

For the appliance example, check the rule decision first, then the actual request:

  1. Save the starting state. Copy the live robots file and note relevant Search Console and firewall settings. Record your proposed change and its date.
  2. Test the full proposed file. Paste the proposed text into a robots rules tester, then enter each agent and full URL using the paths below. Check the entire file, including existing bot-specific groups. Resolve unexpected decisions before replacing the live file.
  3. Verify the published file. Reopen /robots.txt and confirm that it returns the new text successfully, not a cached old file or an HTML challenge. OpenAI documents roughly 24 hours for search systems to reflect a robots update; this is not a citation deadline.
  4. Inspect actual visits. In hosting access logs or firewall events, filter by agent and URL. Record the time, source IP, response status and any challenge or block action. Check the IP against the provider's current published ranges; the agent name alone can be copied by another program.
AgentPath to testExpected rule decision
OAI-SearchBot/guides/oven-installation/Allowed
OAI-SearchBot/cart/Disallowed
GPTBot/guides/oven-installation/Disallowed

These are expected decisions for the sample, not measured crawler visits. Test your real paths too: a rule ending in /cart/ does not match a separate /cart URL.

If a verified, permitted search bot receives 403 Forbidden, inspect the block reason. For 429 Too Many Requests, review rate limits. A 200 OK should return the intended content, not just a challenge page. Adjust the relevant rule rather than disabling the whole firewall.

If no matching request appears, record access as unverified; an absent citation does not diagnose the cause. Once access is confirmed, use the crawling and indexing guide to investigate page eligibility.

Frequently Asked Questions

If I Block a Training Bot, Will I Disappear From AI Search?

Not automatically. OpenAI documents independent search and training controls. Blocking its search bot, however, excludes a site from ChatGPT search answers, though navigational links may remain. Check the purpose of the agent you are blocking.

Does Blocking Google-Extended Remove My Site From Google Search?

No. Google says that token does not affect Search inclusion or ranking. It does cover specified Gemini grounding as well as training, so treating it as a training-only switch would miss part of its scope.

Can One Robots.txt Rule Block Every AI Use of My Content?

No. User-agent: * is a general fallback, not an “all AI” category. Providers and user-triggered fetchers differ, and a current rule cannot reverse previous data use.

Share This Article