Get the plugin
Technical

robots.txt for AI crawlers: what to allow and what to block

Published October 7, 2026 · 6 min read

Short answer: if you want AI answers to mention your site, do not block the crawlers those answers are built from. Training is a separate switch, and you can flip it off on its own. The trap we see most often: a security plugin or Cloudflare rule blocks everything, and the owner never notices because Google still works fine.

When we started auditing WordPress sites for AI visibility, the first thing we checked was robots.txt. It is a five-minute check, and it is still the most common place sites quietly lock themselves out of ChatGPT, Perplexity or Claude answers. This post is the checklist we run.

Search crawlers and training crawlers are different bots

People talk about "AI crawlers" like there is one. There are at least two per company, and they do different jobs:

OpenAI documents the split clearly in its crawler reference: blocking GPTBot keeps your text out of training, but you can still allow OAI-SearchBot so ChatGPT search can show your pages. If you want the traffic, that combination is usually the right starting point.

Google is the odd one out

Google does not have a separate "AI Overviews bot". AI Overviews and AI Mode are part of Search, so they ride on the normal Googlebot crawl. If you block Googlebot, you disappear from classic search too. Nobody wants that.

What Google gives you instead is control over how much text a page gives back: nosnippet, data-nosnippet, max-snippet and noindex. Google-Extended exists, but it covers other uses like training and grounding, not AI Overviews. Google explains the split in its AI features documentation.

A starting file we would not be embarrassed to ship

Here is the shape we use as a baseline. It lets answer engines read the site and opts out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Is blocking training the right call? That is your decision, not ours. Some owners want their text everywhere, models included. Others do not. What matters is that you made the choice on purpose instead of inheriting whatever your host shipped.

The part everyone skips: robots.txt is not the only gate

A clean robots.txt means nothing if something else blocks the bot first. In audits we keep finding the same three culprits:

  1. A security plugin. Wordfence and similar tools ship bot-blocking rules. Some treat unknown crawlers as hostile by default.
  2. The CDN or host. Cloudflare's bot fight mode, or a host-level firewall, can return 403 before WordPress even sees the request.
  3. A staging leftover. Sites cloned from staging sometimes keep "Discourage search engines" checked in Settings > Reading, which adds a sitewide disallow.

Open yoursite.com/robots.txt in a browser and read it with fresh eyes. If a line in there surprises you, find out which plugin wrote it before you delete anything.

Re-check every few months

Bot names change. Companies add new user agents, rename old ones, and split products into new crawlers. The OpenAI publisher FAQ and each vendor's docs are the source of truth, not any blog post, including this one. We re-read them every few months and adjust. You should too.

FAQ

Will blocking training crawlers hurt my rankings? There is no evidence it hurts classic search rankings. It only affects whether your text may be used to train models.

Is robots.txt enforceable? No. It is a voluntary convention. Well-run companies honor it; scrapers ignore it. Treat it as a statement of preference, not a lock.

Do I need a rule for every bot? No. Anything not named falls through to your User-agent: * rules. Name only the bots where you want a different answer.