Chapter 6 of 14

Letting the Robots In

A perfect page that a crawler cannot fetch scores zero where it counts. This chapter is about permission — who is allowed to read you, who thinks they are, and the two traps that silently block the engines you most wanted.

The problem

Everything so far has been about the page. This chapter is about whether the page is reachable, and it is placed here deliberately: access is worth more than quality, because quality is only evaluated after access succeeds. A brilliant page behind a 403 is a page that does not exist.

And access has become genuinely confusing, because “AI crawler” is not one thing. There are at least three kinds, they belong to the same companies, and they want different things:

  • Answer crawlers fetch a page because a user just asked something and the engine is composing a reply now. Blocking these removes you from answers.
  • Training crawlers collect text to train future models. Blocking these is a reasonable position with no effect on today's answers.
  • Search crawlers build the index the answer engines retrieve from. Blocking these removes you from the pool before any question is asked.

The hard way

In 2023 a great many site owners added this, in good faith, after reading an article about AI and copyright: 

User-agent: GPTBot
Disallow: /

That rule blocks OpenAI's training crawler. It does not block OAI-SearchBot, which is what fetches pages for ChatGPT's search. Many people believe they have opted out of AI entirely and have opted out of nothing that affects them. Others copied a stricter snippet that blocked both, and have been wondering for a year why ChatGPT never cites them.

Neither group can tell which one they are in by reading their own robots.txt, because the file is a list of tokens and the tokens do not say what they do.

What Sjá shows you

The AI crawlers view resolves your robots.txt per crawler and states the outcome in words: which named crawlers are allowed, which are blocked, and what each one is for. Not the file — the consequence.

Freddy's result, on the first run: 

GPTBot           blocked     training
OAI-SearchBot    blocked     answers   <- this is the problem
ClaudeBot        allowed     answers
PerplexityBot    allowed     answers
Google-Extended  blocked     training

One line explains three months of ChatGPT never mentioning Freddy. It was never a content problem.

Fetching as a crawler

Here is the trap that robots.txt cannot show you. Sjá can fetch your page as a given crawler — sending that user agent — and report what actually comes back.

This matters because permission is granted in two places and only one of them is your file. Your CDN or firewall has its own opinion about non-browser traffic, and a default bot-protection rule will return 403 to ClaudeBot while your robots.txt says, sincerely, that ClaudeBot is welcome. You have invited a guest your doorman is turning away, and nothing in your own configuration reveals it. 

as ClaudeBot       403 Forbidden     (Cloudflare bot fight mode)
as PerplexityBot   200 OK            18 KB, 0 words of body text
as OAI-SearchBot   200 OK            18 KB, 1,240 words

The middle line is the second trap. 200 OK with no text means the page was served, and everything in it arrives via JavaScript. A crawler that does not execute JavaScript — which includes several answer crawlers — received an empty shell and moved on. Your page is not blocked; it is blank.

This is why chapter 2 said measure as served. The gap between as-served and as-rendered is exactly this trap, visible one chapter before you knew what it meant. If the two differ, fix it by server-rendering the content that matters — not by hoping the crawler renders.

What to actually do

  1. Decide the training question on its merits. Blocking training crawlers is a legitimate choice and costs you nothing in answers. Make it deliberately.
  2. Let answer and search crawlers in. If you want to be cited, this is not optional; it is the whole mechanism.
  3. Fetch as each one and confirm. Your file is an intention; the fetch is the fact.
  4. Fix empty bodies at the server. If the text only exists after JavaScript, it does not exist for a meaningful share of your audience.

What you learned

  • Access outranks quality. Quality is only evaluated after the fetch succeeds.
  • 'AI crawler' is three different jobs — answers, training, search — often from the same company.
  • Blocking GPTBot is not blocking OAI-SearchBot. Most people who think they opted out did not, and some who did never meant to.
  • Fetch as the crawler. Your CDN has an opinion your robots.txt cannot see.
  • 200 OK with no text is worse than a block, because it looks fine from every angle except the one that matters.
Next: in Chapter 7 you answer the question an engine asks before it cites anybody: who is behind this page, and can I resolve them to something real?