Can ChatGPT read your website? The thirty-second check.

Type your own web address into a browser, add /robots.txt to the end, and press return. What comes back decides whether ChatGPT, Claude, Perplexity and Google are allowed to read your site when somebody asks them for a recommendation. It takes thirty seconds and most business owners have never done it.

What is a robots.txt file?

A robots.txt file is a plain text instruction sheet that sits at the root of a website and tells automated systems which parts of the site they are permitted to read. Every website can have one. Many have one without the owner knowing.

It is not a security measure and it hides nothing from people. A determined person can ignore it entirely. But the automated systems that matter here — the ones belonging to Google, OpenAI, Anthropic and Perplexity — check it before reading a site and follow what it says.

The file is short and readable. A typical entry looks like this:

User-agent: OAI-SearchBot
Allow: /

The first line names the system. The second line states what it may read. A forward slash means the whole site. The word to watch for is Disallow.

Which crawlers matter for AI search?

Four matter most. If any of these carries a Disallow instruction, that system cannot read your website when it answers a question.

Crawler	Belongs to	What it affects
OAI-SearchBot	OpenAI	Whether ChatGPT can read your site when searching
Claude-SearchBot	Anthropic	Whether Claude can read your site when searching
PerplexityBot	Perplexity	Whether Perplexity can cite your site
Googlebot	Google	Ordinary search, AI Overviews and AI Mode

There is one more line worth reading carefully. The asterisk in User-agent: * means every system not named individually. If that entry is followed by Disallow: /, the entire site is closed to everything.

What does a blocked crawler actually mean?

It means that when somebody asks that system for a recommendation, your website is not among the things it can look at. Not your services, not your qualifications, not your suburb, not your opening hours.

This is worth sitting with, because it is easy to underestimate. A practice can have a clear, well-written website explaining exactly what it does and who it treats, and none of that is available to a system that has been told not to read it. The work was done. The door was closed.

Everything on your own website has a ceiling, and a blocked crawler sets that ceiling at zero.

Why would my website block them without me knowing?

Because the decision is often made by software rather than by a person. Several hosting platforms, security plugins and content management systems add restrictions to robots.txt by default, and the reasoning is not unreasonable — some of it comes from a period when businesses were actively encouraged to block AI systems.

What tends to happen in practice is that a setting labelled something like "block AI bots" gets switched on once, often by a web developer acting on a general instruction, and nobody revisits it. The distinction between a system that reads your site to recommend you and a system that reads your site to train a model is not usually explained at that moment.

What is the difference between a search crawler and a training crawler?

A search crawler reads a website at the moment someone asks a question, so it can include the site in an answer. A training crawler collects text used to build future versions of a model. They are different systems with different names, and blocking one does not block the other.

Search crawlers	Training crawlers
OAI-SearchBot	GPTBot
Claude-SearchBot	ClaudeBot
PerplexityBot	CCBot
Googlebot	Google-Extended

Allowing search while blocking training is a coherent and defensible position. A business may reasonably want to be recommended without contributing its writing to a training set. The point is that this should be a decision somebody made, rather than a default somebody inherited.

What should my robots.txt say?

For most small businesses that want to be found, the answer is that the four search crawlers should be allowed. The simplest version of that is a single wildcard entry permitting everything, which covers them all:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Naming the crawlers individually achieves the same outcome and makes the decision visible to anyone who looks. Either works. What matters is that nothing important is disallowed.

The final line points to your sitemap, which is a separate file listing your pages. It is not required, but it helps.

What if my website has no robots.txt file at all?

If no robots.txt file exists, nothing is blocked. An absent file is not a problem in itself — it simply means no instructions have been given, and every crawler is free to read the site.

You will usually see a "404 not found" page or an error message. That is a perfectly acceptable result.

What if I find something blocked?

Send the file to whoever manages your website and ask why that instruction is there. It is a two-line change to remove, and there may be a good reason for it that is worth hearing before anything is altered.

Do not edit the file yourself unless you are confident. Robots.txt is unforgiving about spelling and punctuation, and a mistake in it can close more than you intended.

Is this the only thing that matters?

No, and it would be misleading to suggest otherwise. An open robots.txt file does not make a business appear in AI answers. It only means the door is not locked.

Whether a business is actually named in an answer depends on a great deal else — whether the site explains clearly what the business does and who it is for, whether the same description appears consistently across every platform where the business is listed, whether other websites mention it, and whether there are reviews. Most of the mentions AI systems make are drawn from sources the business does not own at all.

But those things are slow, and this one is fast. Thirty seconds, no cost, and a definite answer. It is a reasonable place to begin.

Crawler names verified against the published documentation of OpenAI, Anthropic, Perplexity and Google on 4 August 2026. These names are set by the companies that operate them and do change. Verify current names before relying on any list, including this one.