robots.txt for AI Crawlers: Provider-Specific Controls and Limits

Published May 14, 2026 · ClickRadius Institute

A robots.txt file can express path-level crawl rules for a named token. It cannot establish whether a request reached the origin, whether the requester honored the rule, or whether a provider indexed, retrieved, cited, trained on, or recommended the content. Current AI-associated agents also have different provider-defined purposes and different control limits, so there is no responsible one-size-fits-all AI allowlist.

AI-access policy evidence boundary. This guide documents RFC 9309 matching and provider-stated controls reviewed on September 13, 2026. Permission does not prove provider adherence, crawling, indexing, retrieval, ranking, citation, or training use. A restriction proves what the published policy requests at the audited path, not every possible product route or downstream outcome.

RFC 9309: the matching rule that changes the answer

The Robots Exclusion Protocol is standardized in RFC 9309. A host serves the file at /robots.txt. Each group contains one or more User-agent lines followed by Allow and Disallow path rules.

• Agent-group selection comes first. The crawler selects the group or combined groups with the most specific matching user-agent token. It uses the wildcard group only when no more specific group matches.
• Matching named groups do not inherit rules from User-agent: *. If the wildcard group protects /admin/ and a new named group says only Allow: /, that named agent is no longer governed by the wildcard restriction.
• Path specificity comes next. The longest matching path wins. When an Allow and Disallow rule are equally specific, Allow wins.

That group-selection behavior makes generic named-agent allow snippets unsafe. Before adding a named group, reconstruct the complete intended policy for that agent and test every protected and public path. If the wildcard group already permits public pages, no named Allow: / group is needed.

Provider roles and control limits

OpenAI

OpenAI documents GPTBot for potential training use and OAI-SearchBot for automatic ChatGPT Search crawling as independent controls. It documents ChatGPT-User for some user-initiated actions, says robots.txt may not apply to those requests, and says that token does not determine Search appearance. A rule for one token does not silently control the other two.

Anthropic

Anthropic documents ClaudeBot for content that may contribute to future model training, Claude-SearchBot for search indexing and optimization, and Claude-User for access initiated by a user's request. Anthropic describes independent robots.txt controls for all three.

Perplexity

Perplexity documents PerplexityBot for automatic search surfacing and linking and says it respects robots.txt. Perplexity-User is associated with a user action and generally ignores robots.txt. A PerplexityBot restriction therefore does not prove every user-requested fetch or every possible link is unavailable.

Google, Apple, and Common Crawl

Google documents Google-Extended as a product control token, not a separate request user agent. It controls specified Gemini training and grounding uses and does not affect Google Search inclusion or ranking. Googlebot is the separate ordinary Search crawler.

Apple documents Applebot-Extended as a non-fetching control over specified generative-model use of data crawled by Applebot; restricting it need not block Applebot discovery for Apple search products. Common Crawl documents CCBot as the crawler for its open-web corpus and states that it honors robots.txt. Dataset collection is a separate decision from a provider's automatic answer-search crawler.

No universal named-agent Allow template is safe

Start with the site's existing wildcard policy and sitemap rather than replacing them:

# Existing site policy — example paths only
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

If that policy already permits the public site, an absent named group is not a block. Adding an explicit root allow can discard the private-path protections for the matching agent.

A site that separately decides to restrict the documented training and dataset routes can add root restrictions without claiming they control automatic AI search:

# Optional training and dataset restrictions
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Evaluate Google-Extended separately because Google combines training and grounding under that token:

# Optional Gemini training-and-grounding restriction
User-agent: Google-Extended
Disallow: /

If the policy decision is to restrict documented automatic AI-search crawling, name only those routes and state the consequence narrowly:

# Optional automatic AI-search crawler restrictions
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /

Those examples intentionally omit ChatGPT-User and Perplexity-User because their providers do not present robots.txt as a reliable site-owner control for those user-initiated requests. They also omit Googlebot and bingbot because an AI-specific policy should not silently change ordinary search crawling. Do not add or remove any block until the owner has made the corresponding content-use and discovery decision.

Content-Usage is a draft preference, not access control

The current IETF attachment draft and vocabulary draft define a structured Content-Usage field with current categories including train-ai and search. For example:

Content-Usage: train-ai=n

These are active IETF Internet-Drafts, not finalized consensus standards and not a provider-enforcement guarantee. The current search category concerns linked search results and excludes generative summaries; the vocabulary does not define a rag category. A published preference does not grant network access and does not prove that a recipient recognized or followed it.

An llms.txt file is likewise an informational publisher guide, not an RFC 9309 permission layer. It cannot override robots.txt, a CDN, authentication, or an application response, and its presence does not prove discovery or citation.

Audit the effective path, not just the file

1. Fetch the public bytes. Record the host, URL, status, response headers, body hash, and observation time. A repository copy does not prove what the public host served.
2. Evaluate each token and URL under RFC 9309. Test the homepage, representative public pages, and every protected path. Record which named or wildcard group supplied the decision.
3. Reconcile other layers. Inspect CDN, WAF, authentication, application routing, and origin behavior separately. A permissive text file does not establish end-to-end reachability.
4. Verify logged identities. A user-agent string is copyable. Use the provider's current network-verification method before attributing a request, then retain the URL, time, status, and response layer.
5. Record what remains unknown. No request proves compliance; no request absence proves a block; and a verified request does not prove indexing, answer use, a source link, a human click, or training use.

A robots audit can prove the policy bytes served and the RFC decision for a token at a path. Provider behavior and answer outcomes require different evidence.— ClickRadius Institute

Frequently asked questions

Will blocking Google-Extended keep me out of AI Overviews?

Google documents Google-Extended as a product token for Gemini training and grounding. It is not a separate request user agent, and its setting does not affect Google Search inclusion or ranking. That does not make it an AI Overview exclusion switch or prove any particular generated-answer outcome.

Is robots.txt legally enforceable against AI companies?

RFC 9309 standardizes how compliant crawlers interpret the file; the file itself is not authentication or a firewall. This article does not provide a legal conclusion. For technical access enforcement, use authenticated access or verified network controls and test the public response.

Should a small business block AI training crawlers?

There is no universal answer. Separate training, grounding, automatic search, user-initiated access, ordinary search, and dataset collection; then record a policy decision for each documented control. Neither allowing nor blocking a token proves a future model, source-link, traffic, or revenue outcome.

The free AI Readiness Score estimates five weighted input families observed at the submitted URL. It is not an end-to-end provider access test and does not prove an answer outcome. See the current service scope on the pricing page.