Chapter 11 of 14

Did the Crawlers Come?

Chapter 6 was about permission in principle. This is about what happened in fact. Your access log is the only place on the internet that records whether the engines you care about actually came, and almost nobody reads it.

The problem

Everything else in this course is inference. Your robots.txt says ClaudeBot may enter; a fetch as ClaudeBot returns 200; therefore ClaudeBot can read you. Each step is sound and the conclusion is still a prediction — it says the door is open, not that anyone walked through it.

Your server log is different in kind. It is a record of things that happened, written by your own machine, which has no opinion and nothing to sell. For the question “are the answer engines actually reading my site?” it is the only primary source that exists.

The hard way

You can grep a log for user agents. People do, and it misleads them in three ways, all of which matter:

  • Anyone can claim any user agent. A scraper announcing itself as GPTBot is a line in your log that looks exactly like the real thing.
  • Fetching robots.txt is not a visit. A crawler that reads your rules and leaves shows up in a grep as traffic. It read the sign on the door.
  • A 403 looks like an arrival. The request is in the log either way; whether it succeeded is a different column, and the grep does not care.

Evidence, not prediction

Point Sjá at an access log — nginx, Apache, or a Cloudflare export — and it answers the three questions the grep cannot.

Which visits were real. Claimed user agents are checked against the address ranges the crawler operators publish. A request claiming to be GPTBot from an address outside OpenAI's published ranges is reported as unverified — which is not the same as malicious, but is certainly not evidence that OpenAI read your site.

Who only ever read the rules. A crawler whose entire history is /robots.txt did not visit you. It checked whether it was allowed and went away. That distinction is invisible in a raw count and decisive in interpretation.

Who is blocked but keeps visiting. The most useful line in the whole view: 

OAI-SearchBot    417 requests    403 Forbidden    blocked by robots.txt

An engine has been trying to reach you four hundred times and has been turned away every time. It wants your content. Your configuration says no. Chapter 6 found that rule; this is the same finding with a number attached, and the number is what makes people act.

Visits over time

Crawler visits are plotted over time, and the shape carries information that a total does not.

  • A cliff — steady visits, then nothing. Line the date up against your deploy log. Something you shipped that day closed the door: a new CDN rule, a robots change, a redirect loop.
  • A slow decline is usually staleness. Crawlers re-visit at a rate related to how often they find something new.
  • A step up after a deploy is the confirmation you have been waiting six chapters for — the fix landed and the engines noticed.
Why this chapter comes after the probe. Chapter 10 told you whether you are cited. This tells you whether you were ever read. If the answer here is no and the answer there was no, you do not have a content problem — you have a door problem, and no amount of rewriting will fix it. Reading these two chapters in the other order has cost people months.

What you learned

  • Your access log is the only primary source for whether an engine actually read you.
  • Claimed user agents are checked against published address ranges, because anyone can claim to be GPTBot.
  • Reading robots.txt is not a visit. It is checking the sign on the door.
  • 'Blocked but keeps visiting' is the line that makes people act — an engine wants you and cannot have you.
  • A cliff in the graph has a date, and the date is in your deploy log.
  • Not cited plus never read is a door problem, not a content problem.
Next: in Chapter 12 you make all of it keep working: monitors on a schedule, runs that happen while the app is closed, and alerts when a number moves the wrong way.