Why it matters
When someone asks ChatGPT or Claude for a recommendation, the assistant searches the web and reads the pages it finds. A page its crawler isn’t allowed to fetch can’t be read, so it can’t be quoted, and the business behind it is easier to leave out.
Many sites block AI crawlers without meaning to: a robots.txt template copied from somewhere, a security plugin, or a hosting setting switched on for everyone. The site still works for people and for Google, so nobody notices.
What the checker looks at
- robots.txt The rules for each crawler by name, or the rules for all crawlers (*) when it isn’t named, read the way the robots.txt standard (RFC 9309) says: the most specific rule wins, Allow wins a tie.
- The server’s answer Your home page is requested with each crawler’s own user agent. A 403, 401, 429 or 503 to the crawler, while the same page loads for a browser, means something on the server turns it away.
- Cloudflare If your site runs behind Cloudflare, its bot settings can block AI crawlers that robots.txt allows. The checker says when your site answers through Cloudflare.
The crawlers it checks
| Crawler | Assistant | What it does |
|---|---|---|
| OAI-SearchBot | ChatGPT | Finds and reads pages for ChatGPT’s search answers. |
| ChatGPT-User | ChatGPT | Fetches a page when a user’s question needs it. |
| GPTBot | ChatGPT | Collects text for training OpenAI’s models. |
| Claude-SearchBot | Claude | Finds and reads pages for Claude’s search answers. |
| Claude-User | Claude | Fetches a page when a user’s question needs it. |
| ClaudeBot | Claude | Collects text for training Anthropic’s models. |
| PerplexityBot | Perplexity | Finds and reads pages for Perplexity’s answers. |
| Googlebot | Google Search, and with it AI Overviews and Gemini’s search. | |
| Google-Extended | A robots.txt name only: whether Google may use your pages to train Gemini. Googlebot does the fetching, so it isn’t requested. |
How to let them in
Find the rule
Open yoursite.com/robots.txt. Look for a User-agent line naming the crawler, or a Disallow: / under User-agent: *.
Allow the search crawlers
Add a group such as “User-agent: OAI-SearchBot”, “User-agent: Claude-SearchBot” and “User-agent: PerplexityBot” followed by “Allow: /”. Keep your other rules as they are.
Check the server
If robots.txt allows a crawler but the checker shows a refusal, look at your firewall, security plugin or host. On Cloudflare, see Security → Bots and the AI crawler settings.
Check again
Run the checker once more. Crawlers read robots.txt again within a day or so.
Frequently asked questions
Does blocking GPTBot keep me out of ChatGPT?
Not on its own: GPTBot collects training data. ChatGPT’s search answers read pages with OAI-SearchBot and ChatGPT-User. Blocking those keeps your pages out of what ChatGPT reads when it answers.
My site has no robots.txt. Is that a problem?
No. Without a robots.txt every crawler is allowed. The checker says so.
Why does the checker show a refusal when robots.txt allows the crawler?
Something on the server answered the crawler’s user agent with an error while the same page loaded for us: a firewall, a security plugin, or a hosting or Cloudflare bot setting.
Is it free?
Yes, and there is no sign-up. It only reads your robots.txt and your home page.