Should You Let AI Crawlers Read Your Site?

There is a genuine debate here and most of the loud end of it does not apply to you. The people arguing hardest for blocking AI crawlers are publishers, and they are right for reasons that do not transfer to a business that fits bathrooms.

The first job is finding out what you already do

Most businesses blocking AI crawlers never decided to. A host added it, a plugin added it, a template shipped with it, or a previous developer read an article in 2024.

Open a browser and go to your domain followed by /robots.txt. You are looking for lines like:

User-agent: GPTBot
Disallow: /

The names worth knowing:

Agent	Belongs to	What it does
GPTBot	OpenAI	Crawls for training
OAI-SearchBot	OpenAI	Crawls so ChatGPT can cite live pages
ClaudeBot	Anthropic	Crawls for training
PerplexityBot	Perplexity	Crawls for its answer engine
Google-Extended	Google	AI training and some AI features, not Search

OpenAI documents its own in a bots guide, which is worth reading if you want the current list rather than the one in whatever article your developer read.

If any of those are disallowed, that is a decision somebody made on your behalf. It may be the right one. It is worth knowing it exists.

What blocking actually does

It stops a well-behaved crawler reading your public pages. That is all.

It is not a security measure. robots.txt is a request, honoured by companies that choose to honour it. Anything genuinely malicious ignores the file entirely. If somebody has sold you crawler blocking as protection, they have sold you a sign that says please do not.

What it does achieve is absence. If the tool cannot read your site, it has nothing of yours to draw on when somebody asks it who to call.

The argument for blocking, stated fairly

It is a real argument and it goes like this: these companies built products worth a great deal by reading work they did not pay for, they now answer questions people used to answer by visiting your site, and you get nothing back.

That is true, and for a publisher it is existential. If your business is selling the writing, an AI that summarises your writing for free has taken the thing you sell.

Why it does not apply to a trade business

Your product is not the words. Nobody is going to read an AI summary of your page about blocked drains and conclude they no longer need the drain cleared.

For you the exchange is different: they read a page you published publicly to attract customers, and in return your name appears in front of somebody deciding who to call. That is the same bargain you have made with search engines for twenty years and it has worked out fine.

Blocking does not protect revenue here. It removes you from a surface that increasingly decides who gets contacted.

There is one edge case worth naming. If you have published genuinely original material that is itself the product, a set of technical guides you sell access to, or photography somebody licenses, that is a publisher's position and the publisher's argument applies. That is rare in the trades and it is not what most people blocking are protecting.

Google-Extended is the one people get wrong

Google separates these. Googlebot crawls for Search. Google-Extended is a separate control covering AI training and certain AI features.

Blocking Google-Extended does not remove you from ordinary Search results. Blocking Googlebot does, completely, and that mistake does get made. If you are going to change anything in robots.txt, be certain which agent you are naming.

A related trap: a staging site that was set to disallow everything, then went live without anybody changing it back. That blocks the lot, Googlebot included, and it is the most costly thing anybody has ever forgotten to delete. If your site launched recently and nothing is being indexed, check this before anything else.

What I would actually do

For nearly every trade, contractor and local service business: let them in. The default, which is saying nothing about them at all, is the right setting. You do not need to add anything to welcome them.

Then make sure there is something worth reading when they arrive, which is the part that actually matters and the part nobody talks about. A crawler with full access to a site that says nothing specific about what you do, where, and for how much will produce an answer about somebody else.

How to tell whether it is making any difference

Open your robots file and confirm the door is open. Then go and ask the tools, in a signed-out window, the questions a customer would ask: who would you recommend for this trade in this town, what does this job cost here, who does emergency work in this area.

Write down what comes back and do it again in a month. Nothing reports how often you were mentioned, so a note you keep yourself is the only measurement that exists here.

If your name never appears, the robots file is almost never the reason. It is far more often that there is nothing specific on your site to mention, which is a separate job and the one worth the money. The door being shut is just the version of the problem you can fix in five minutes.