To appear in ChatGPT search, allow the OAI-SearchBot user agent in robots.txt. To appear in Perplexity results, allow PerplexityBot. GPTBot is a separate bot that OpenAI uses for model training, so you can block it without leaving ChatGPT search. Changes can take about 24 hours to apply.
Which AI bots matter for visibility?
Each AI company runs more than one bot, and each one does a different job. Blocking the wrong one is the most common way a site disappears from AI answers by accident. The two companies below publish their user agents in official documentation.
| Bot | Company | What it does | Allow it to appear in answers? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | Yes |
| GPTBot | OpenAI | Crawls content that may be used to train models | No, this is a training choice |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT about it | Not controlled by robots.txt alone |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results | Yes |
| Perplexity-User | Perplexity | Fetches a page when a user asks a question | Generally ignores robots.txt |
Sources: OpenAI's bots documentation and Perplexity's bots guide.
What should my robots.txt say?
If you want to be cited in both products, make sure nothing blocks their search bots. A site with no rules for them is already allowed. You only need explicit lines if a broad rule elsewhere in the file blocks them. This example allows both search bots and opts out of OpenAI training:
User-agent: OAI-SearchBotAllow: /
User-agent: PerplexityBotAllow: /
User-agent: GPTBotDisallow: /
OpenAI states that each bot's setting is independent, so this combination is valid. It also notes that if you allow both OAI-SearchBot and GPTBot, it may use a single crawl for both purposes.
Does blocking GPTBot hide me from ChatGPT?
Not from search. OpenAI says GPTBot is for content that may be used to train its foundation models, and that blocking it signals your content should not be used for training. Appearing in ChatGPT's search features is governed by OAI-SearchBot. Blocking OAI-SearchBot has the opposite effect: your site will not appear in ChatGPT search answers, though it can still show up as a navigational link.
Whether to allow training is a business decision. Allowing it is not required for visibility.
Why robots.txt is not the whole story
OpenAI says that to appear in search you should also allow requests from its published IP ranges. A firewall, CDN rule or bot-protection setting can block a bot that robots.txt welcomes. The file looks right and the site is still invisible.
User-triggered visits behave differently. OpenAI says ChatGPT-User visits happen when a person asks a question, so robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. If you need to restrict those visits, you need controls at the server level, not a robots.txt line.
How to check that it worked
- Open
yourdomain.com/robots.txtand read it top to bottom. Look for aUser-agent: *block withDisallow: /, which blocks everything not named elsewhere. - Check your CDN or firewall for rules that challenge or block unknown bots, and compare against the IP ranges OpenAI publishes.
- Wait about a day. OpenAI says search systems can take around 24 hours to adjust, and Perplexity says up to 24 hours as well.
- Ask each product a question your page answers and see whether your domain is cited. A single test proves little, so repeat it over a few days.
Nightcrop's audit includes a check that AI crawlers can reach your site, which catches the firewall case that a robots.txt read misses.
What about Google?
Google's AI features use the same Search index. Google says a page must be indexed and eligible to appear in Search with a snippet, with no extra technical requirements, as described in its AI features documentation. So there is no separate AI bot to allow for that. Plain Googlebot access is what counts. For other optional files, see whether you need an llms.txt file.
A common mistake to avoid
Many sites copy a "block all AI bots" snippet from a forum after reading about training data. These snippets often list every AI user agent in one block, including the search bots. The result is a site that has opted out of training and out of being cited at the same time.
Before pasting any list, look up each user agent in the company's own documentation, decide separately whether it is for search or for training, and only block the ones whose purpose you object to.
Keep a dated note of when you changed the file and which bots you allowed. If your citations change later, that note tells you whether your own edit is a possible cause or whether to look at your content instead.
Frequently Asked Questions
Which user agent gets me into ChatGPT search?
OAI-SearchBot. OpenAI says sites that opt out of it will not appear in ChatGPT search answers.
Can I block AI training but stay visible?
For OpenAI, yes. Block GPTBot and allow OAI-SearchBot. Perplexity says PerplexityBot is not used to crawl content for AI foundation models.
Does Perplexity obey robots.txt?
PerplexityBot does. Perplexity-User, which fetches pages for a person's question, generally ignores it.
How long until a robots.txt change takes effect?
OpenAI says around 24 hours for search. Perplexity says up to 24 hours.
My robots.txt allows the bots, so why am I not cited?
Allowed is not the same as chosen. Check firewall rules, then check that your page answers the question clearly.