AI Crawlers and robots.txt: Which Bots to Allow for AI Search

Allow the crawlers that find and cite your pages, and decide separately about the ones that collect training data. For AI search that means letting OAI-SearchBot, PerplexityBot and Claude-SearchBot in, while GPTBot, ClaudeBot and Google-Extended are a choice you make about training. This guide shows the safe robots.txt for each case.

A robots.txt file is a plain text file at the root of your site that tells crawlers which paths they may fetch. Since 2023 the major AI companies have added their own named bots to it, and each one does a different job. Mix them up and you can switch off your visibility in ChatGPT or Perplexity without meaning to.

This post covers one technical step of generative engine optimization: crawler access. It does not replace good content, but a blocked bot cannot read even the best page.

Which AI crawlers should you allow in robots.txt?

Allow the bots that find and cite your pages: OAI-SearchBot, PerplexityBot, Claude-SearchBot and ordinary Googlebot. Decide separately on training bots such as GPTBot, ClaudeBot and Google-Extended. Each vendor treats these as independent settings, so you can appear in AI answers while opting out of model training.

OpenAI says each of its settings is independent, and that you can allow OAI-SearchBot to appear in search while disallowing GPTBot (OpenAI crawler documentation). That split is the core idea. Treat “search”, “fetch on request” and “training” as three separate decisions.

What each AI crawler actually does

The names look alike, but the jobs differ. This table sums up the bots the vendors document.

Perplexity states that PerplexityBot is “not used to crawl content for AI foundation models” and exists to surface and link sites in its results. Its second bot, Perplexity-User, acts only when a person asks a question, so it generally ignores robots.txt rules (Perplexity bot documentation). OpenAI says robots.txt rules may not apply to ChatGPT-User either.

Anthropic lists three bots: ClaudeBot for training data, Claude-User for questions that need a live fetch, and Claude-SearchBot for search quality. It says they honour robots.txt, and that disabling the user or search bots may reduce your visibility in Claude (Anthropic crawler help article).

Google-Extended is a token, not a crawler

Google-Extended has no user agent string of its own. It is a robots.txt token that controls whether content Google crawls can train future Gemini models and be used for grounding in Gemini Apps. Google says it “does not impact a site’s inclusion in Google Search” (Google crawler documentation). Googlebot stays untouched.

Should you block GPTBot and ClaudeBot for training?

Blocking GPTBot tells OpenAI not to use your content to train its models. It does not remove you from ChatGPT search, because OAI-SearchBot handles that and is a separate setting. Most small businesses with public marketing pages lose little by allowing training bots, but the choice is yours.

Reasons to block are real for publishers, paywalled sites and anyone with original data they sell. A restaurant, clinic or estate agency is in a different position. Your goal is to be named when someone asks an AI tool for a recommendation, and the more of your public facts the models can read, the better the odds.

From our work: a clean technical score does not tell you whether AI bots are let in. On the Baku Lounge website, a first site for a Wrocław restaurant that had only Instagram and a Google Maps listing before, Google PageSpeed Insights gave an SEO score of 100 on 21 October 2026. That score says nothing about GPTBot or PerplexityBot, so robots.txt needs its own check.

A safe robots.txt example for AI search

The version below allows the search bots and opts out of training. Paste each line on its own line in the file at yourdomain.com/robots.txt, keeping the pairs together. To allow training as well, change the three Disallow lines to Allow: /.

User-agent: OAI-SearchBot

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: Claude-SearchBot

Allow: /

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: Google-Extended

Disallow: /

A bot follows the most specific group that names it. If your file already has a general User-agent: * group with Disallow rules for folders such as /admin/, a named group does not inherit them, so repeat any rule you still want. Keep your sitemap line in place as well.

How do you check that AI bots can reach your site?

Open yourdomain.com/robots.txt in a browser and read every group for the named bots. Then request a page with each bot’s user agent using curl and compare the response codes with a normal visit. Finally, look for those names in your server logs to see whether they arrive at all.

Do the curl test, because a correct robots.txt is only half the picture. The next section explains why.

Why robots.txt may not be the only thing blocking AI bots

Robots.txt is a request, not a lock. Google describes it as not a mechanism for keeping a page out of Google (Google robots.txt introduction), and AI crawlers follow it voluntarily. The same applies in reverse: a clean file does not guarantee access.

Firewalls, CDN bot settings and security plugins often block unfamiliar user agents before robots.txt is read. Perplexity publishes the IP ranges for both its bots and recommends allowing them alongside the robots.txt rule. Anthropic advises against blocking its IP addresses, since that can stop its bots reading your robots.txt at all.

If you want to see whether any of this changes who finds you, our AI visibility tracking page explains how we measure it. For the wider technical basics, read our post on technical SEO errors that cost traffic, and see how the same ideas apply in our AI search guide for estate agents. The SEO for ChatGPT page covers what happens after the bot gets in.

Frequently asked questions

Get your AI search setup checked

If you are not sure which bots your site lets in, we can read your robots.txt, test each crawler and fix what blocks them. Contact LineUp and we will tell you what we find.