OAI-SearchBot robots.txt: the setting that controls ChatGPT Search

The first AI visibility check is not a prompt

A lot of AI-search advice starts with schema, content formats or prompts. There is a more basic check: can the search crawler reach the site at all?

OpenAI documents OAI-SearchBot specifically for ChatGPT Search. If a site blocks it in robots.txt, OpenAI says the site's pages will not be shown in ChatGPT search answers, apart from limited navigational links. That is a documented eligibility control, not a theory about ranking.

OAI-SearchBot is not GPTBot

OpenAI separates search visibility from model training. OAI-SearchBot is the Search crawler. GPTBot is the crawler OpenAI uses for content that may be used to improve its generative AI foundation models.

ChatGPT-User is a third agent. OpenAI uses it when a person asks ChatGPT to open a page. It is not the control for appearing in ChatGPT Search, and it is not a training crawler. Blocking it can stop that user-directed fetch. It does not, by itself, decide Search eligibility.

That means a publisher can allow OAI-SearchBot while choosing a different policy for GPTBot. Blocking training does not have to mean blocking ChatGPT Search.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Claude has separate controls too

Anthropic documents the same basic separation with different user agents. Claude-SearchBot is used for search indexing, Claude-User supports user-directed retrieval, and ClaudeBot is used for model-development crawling.

For a business that wants to be discoverable in Claude without treating every kind of crawler the same, those distinctions matter.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

robots.txt is necessary, but it is not the whole job

An allow rule does not guarantee a recommendation. It only removes one technical barrier. The page still has to be reachable through the CDN or firewall, indexable where appropriate, relevant to the question, and supported by clear public facts.

For OpenAI specifically, the company also recommends allowing requests from its published SearchBot IP ranges. A robots.txt file that says yes while a firewall returns 403 is still a closed door.

What businesses should check

1. Open /robots.txt and look specifically for OAI-SearchBot, Claude-SearchBot and Claude-User rules.
2. Separate search/retrieval policy from training policy. Do not treat GPTBot and ClaudeBot as if they are the same thing as the search crawlers.
3. Check CDN, WAF and bot-protection rules for accidental 403s or challenges that automated crawlers cannot pass.
4. After changing robots.txt, retest. OpenAI says its systems may take about 24 hours to adjust to a robots.txt change.
5. Then test actual customer questions and record the answers and sources. Crawl eligibility is the start of the evidence chain, not the finish.

Questions