Sjá 0.6.0: GEO along the path a page actually travels

There's a comfortable fiction at the heart of most "AI visibility" tooling, and until last week Sjá told it too. The fiction goes: an answer engine fetched your page, read all of it, understood it, and will quote whatever it likes. Score the page for schema, for question headings, for an llms.txt, and you have measured your AI visibility.

Except that isn't what happens. A crawler asks robots.txt whether it may come in. Then it asks the server, and the server — or the CDN in front of it — may say something different. If it gets in, it doesn't read the page; it extracts the main content, keeps what fits a reading budget, and picks passages it can quote. And at the end of all that, a real user asks ChatGPT a real question, and your page either shows up in the answer or it doesn't.

Sjá 0.6.0 rebuilds the GEO category along that path. Three layers — access, content, outcome — fifteen checks instead of four, and roughly half the score under Improve AI visibility, which is where a fresh install now starts. Here's how each layer works, why it's built the way it is, and what it looks like on a real page.

Access: may they come in, and does the door actually open?

The first question is embarrassingly basic and almost nobody checks it: can the answer crawlers fetch this page at all?

Sjá already read robots.txt properly — every user-agent group, most-specific match wins, the way the crawlers themselves resolve it. 0.6.0 adds the check that uses it: the page's own path is tested for OAI-SearchBot, ChatGPT-User, Claude-User and Perplexity-User. Blocking one of these is a fail, weight 10. These are the crawlers that fetch pages to answer questions right now; block them and you are absent from answers today.

Blocking GPTBot, ClaudeBot or CCBot — the training crawlers — is a weight-zero note. That's a licensing decision, and a perfectly respectable one. It is not a visibility problem, and a tool that penalised you for it would be confusing two different things.

But robots.txt is advice. So Sjá now fetches the page a second time, with OAI-SearchBot's real User-Agent, and looks at what comes back. This is the only way to see a CDN's "block AI bots" switch, a WAF rule, or a bot challenge:

  Reachable as an AI crawler          fail   weight 8
    Fetched as OAI-SearchBot: HTTP 200, but the body is a Cloudflare
    challenge ("Just a moment…", cf-mitigated: challenge).
    robots.txt allows this crawler; the edge does not.

Note the status: 200. A naïve check would have passed it. The challenge page is recognised by its fingerprints regardless of the status code, because a crawler that receives a challenge has received nothing.

The fix prompt for this check includes the curl commands that prove the fix — fetch with each crawler's User-Agent and look at what you get — because "I flipped the switch in the dashboard" and "the crawler can now read the page" are different claims.

One more access check, and it's the one that catches modern sites: content without JavaScript. AI crawlers fetch HTML and execute nothing. Fewer than fifty words in the HTML, plus the fingerprints of a client-rendered app — an empty #root or #__next, a <noscript> plea — is a fail. Your beautiful React marketing page may be, to ChatGPT, a blank sheet with a loading spinner.

Content: what the model sees

This is the layer I'm most pleased with, because it makes something visible that was always there and never shown.

Engines don't read pages. They extract. So every Sjá report now carries a panel called What the model sees: the page's text in reading order, with navigation, footers, cookie banners, sidebars and forms stripped the way a readability extractor strips them. The report tells you how many visible words were discarded as boilerplate — and on a lot of sites that number is a quiet shock.

Then it draws a line. At roughly 2,000 tokens (about four characters each — stated as an estimate, because it is one) there's a marker, and every block of text knows whether it starts inside the budget or after it. Your best paragraph, the one that actually answers the question, may well be sitting below the line.

And then it picks passages. The five most quotable, each scored 0–100, with the reasons on both sides:

  Passage 2 · score 78 · starts inside budget
    "Askr embeds PHP in-process and, in worker mode, boots the
     application once and serves many requests. On a 4-core VM that
     removes roughly 30 ms of framework bootstrap per request."
+ 47 words, stands alone, full sentence, carries a figure
+ directly under a heading
− heading is not a question

Forty to ninety words. Stands alone — doesn't open with "This" or "It" or "However" (or Dette and Derfor, since a lot of our users write Norwegian). Sits directly under a heading, doubly so if the heading is a question. Carries a figure or a date. Starts inside the budget.

The reasons on the minus side are the point. The gaps are what a writer changes. Turn the heading into the question the passage answers, move the passage above the fold, and you've done GEO — not by adding a schema block, but by writing the thing an engine can quote.

Three checks fall out of this: Quotable passages, Answer within the reading budget, and Signal to noise. And the MCP server and the CLI return the same extraction, so an assistant sitting in your editor edits the exact text the engine will see.

Structured data, read rather than counted

The old GEO had a check that counted @types. 0.6.0 reads them for what an engine needs.

A block that isn't valid JSON used to register as "no structured data." Now it's a fail that says so — same outcome for the engine, entirely different fix. Required properties per type fail when missing (an Article without a headline, a HowTo without steps); recommended ones warn. FAQPage is checked question by question: a Question without a name or an answer text is called out individually.

Two things engines increasingly lean on, and that almost no tool checks: the publisher entity — is there an Organization behind this page, with a name, a canonical URL, a logo, sameAs links? Or only a WebSite, or nothing? A crawl now lists every distinct Organization name a site claims to be, with page counts, which is a gentle way of discovering that your site says it's three different companies. And dates that agree: datePublished, dateModified, article:* meta, the first <time datetime>, the Last-Modified header — collected and held to consistency. Modified before published is a fail. Two modified dates that disagree, a date in the future, a date over two years old: warnings. No dates at all is a note, weight zero — not everything has a date, and inventing one would be worse.

Outcome: don't predict. Ask.

Everything above is a prediction. Careful, deterministic, explainable — but a prediction. The last layer asks the engines directly.

From a page's report, type the question a user would actually ask — what's the fastest way to run Laravel without PHP-FPM? — and press Probe. Sjá puts it to ChatGPT (Responses API with web search), Claude (Messages API with web search) and Perplexity, concurrently, with your own API keys, and judges each answer:

  Citation probe · "fastest way to run Laravel without PHP-FPM"

ChatGPT      cited     position 2 of 5
Claude       cited     position 1 of 3
Perplexity   not cited
             cited instead: frankenphp.dev, roadrunner.dev, laravel.com

Does it cite your site? Where among the sources? And — this is the column you'll keep looking at — who was cited instead. That's your competitor list, drawn from the only source that matters.

Every probe is stored, failures included, so a gap in the history is visible as a gap rather than papered over. Trends shows the latest verdict per page and engine with the earlier ones as dots.

We were careful about what to say around this. The UI tells you plainly what a probe is: one sample of a non-deterministic system, through APIs that are close to — but not identical with — the consumer products, costing a few cents per engine on your own account. Ask the same question twice and you may get two different answers; that's the system, not a bug. What you should read is the trend over weeks and the competitor list, not one Tuesday's result.

The probe needs keys, one per engine, under Citation probe in Settings. An empty OpenAI or Anthropic key falls back to the key you already gave the AI agent, so nothing gets pasted twice.

And did they actually come?

Sjá could already read your nginx or Apache access log and tell you which AI crawlers really visited. 0.6.0 gives that a time axis: visits per crawler per day, stored per log file so two servers' logs add up and re-analysing a file replaces its days. Did ClaudeBot come back after we changed robots.txt? is now a chart, not a guess.

And because a User-Agent is a string anyone can send, there's Verify addresses. Sjá downloads the IP ranges each operator publishes — OpenAI's three crawlers, Googlebot, Bingbot, PerplexityBot, Applebot — and checks whether every visit claiming to be GPTBot came from OpenAI's ranges, some did (with the offending addresses), or none did. Operators that publish no list, Anthropic among them, are reported as unverifiable — never, ever as verified. The lists are cached locally for a week. No address from your log leaves your machine, because an access log is full of your visitors' IP addresses and that stays true in every feature we build.

Why scores will move

We should say this plainly, because it's the kind of thing that generates support email: your scores will change between 0.5.1 and 0.6.0.

GEO goes from 24 weight points to 80. Pages the crawlers can reach and read will go up. Pages behind a bot-blocking CDN, a JavaScript shell, a broken schema block or a missing publisher entity will go down — sometimes a lot. Compare runs across the upgrade with that in mind.

This is the point. A score that stayed stable while the tool started measuring the right things would be a score that wasn't measuring anything.

Try it on your own site

Update (0.4.2 or later updates itself; two small migrations run on first launch), open a page you care about, and look at three things:

1. Reachable as an AI crawler. If it's red on a 200, you've found the CDN switch nobody remembers flipping.

2. What the model sees. Where's the budget line, and what's below it?

3. Probe the question you most want to be the answer to. Then look at who was cited instead.

Open the report. Read the footnotes.