OAI-SearchBot vs GPTBot: Which OpenAI Crawler Should You Allow?
Quick answer: OAI-SearchBot and GPTBot serve different purposes. Allowing OAI-SearchBot lets OpenAI use a site in ChatGPT search features; blocking it opts the site out of ChatGPT search answers. GPTBot is the training crawler, so blocking GPTBot is a separate choice about use in OpenAI’s generative AI foundation models.
They are often placed in the same “AI bots” bucket. That is the mistake. A blanket robots rule can make a policy decision about training and accidentally remove a site from a search surface the business wants to reach.
The three OpenAI user agents mean different things
| User agent | What it is for | What a robots.txt choice affects |
|---|---|---|
OAI-SearchBot | OpenAI’s search crawler | Whether a site may surface in ChatGPT search results |
GPTBot | Training crawler | Whether crawled content may be used to train OpenAI foundation models |
ChatGPT-User | User-triggered visits and some Custom GPT actions | Not used to decide whether content appears in Search; robots.txt may not apply to user-initiated requests |
OpenAI documents the settings as independent. A publisher can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training. That is the configuration to consider when the goal is search visibility without opting into training use.
A conservative robots.txt example
If this policy matches the organization’s legal and product decisions, the relevant rules are straightforward:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Do not copy that example blindly. A crawler rule is a business and licensing decision, not a universal SEO recommendation. Review your wider robots policy, any CDN or WAF rules, your legal position, and whether the crawlers can reach the pages you actually want found.
Do not use ChatGPT-User as the search control
OpenAI says ChatGPT-User is used for actions initiated by a user, not for automatic web crawling, and is not used to decide whether content appears in Search. OpenAI specifically directs site owners to use OAI-SearchBot for Search opt-outs and automatic crawl management.
That distinction matters when diagnosing logs. A ChatGPT-User visit can be a user asking ChatGPT to open a specific page. It does not prove that the page is eligible for ChatGPT search, just as an OAI-SearchBot request does not guarantee a future citation.
How to audit the change safely
- Save the current robots.txt file and identify the rule that applies to OAI-SearchBot.
- Check for broad
User-agent: *rules, because they may override the intent you thought a later rule expressed. - Confirm your CDN, firewall, and hosting layer are not blocking the crawler after robots.txt allows it.
- Verify crawler identity against OpenAI’s published IP ranges rather than trusting a user-agent string alone.
- Recheck the live file at
/robots.txtafter deployment. OpenAI says it can take roughly a day for its systems to adjust after a robots.txt update. - Measure visibility separately. Crawling access is eligibility, not a promise that a particular answer will cite the site.
One more robots.txt warning
Robots.txt manages crawler access; it is not a reliable way to keep an HTML page out of Google Search. Google recommends noindex, password protection, or removal when the goal is to prevent a page from appearing in search results. Keep access control, indexing control, and AI crawler policy as separate decisions.