Elyra
Blog
Sjá
Sjá · · 6 min read

Before you optimize for AI, check whether the robots even show up

Structured data, llms.txt and quotable paragraphs are worthless if the crawler never arrives. Elyra Sjá 0.5.0 reads your access log to show which AI crawlers actually came, whether your robots.txt lets them in, and where the two disagree.

Before you optimize for AI, check whether the robots even show up

There's a question almost nobody asks before diving into "AI search optimization", and it's the one that decides whether any of the rest matters:

Is anything actually coming to read your site?

We build Elyra Sjá, a desktop tool for SEO and GEO work. For most of its life it answered a different question — is this page legible to an answer engine? Does it have structured data, a sane heading hierarchy, an llms.txt, answer-shaped subheadings. Useful stuff. But all of it is a prediction. It's us guessing how a machine will read a page.

Then a colleague asked the obvious thing: how do we know GPTBot ever came?

We didn't. So we built it.

Why this is the first question, not the fifth

Here's the uncomfortable arithmetic. If your robots.txt blocks the crawler that fetches pages to answer questions, then:

  • your structured data doesn't matter

  • your llms.txt doesn't matter

  • your beautifully quotable paragraphs don't matter

You are invisible, and no amount of on-page work changes it. One line in a text file outranks everything else you might do.

And this isn't rare. Somebody adds a well-meaning rule during a migration, or copies a robots.txt from a template that blocks AI crawlers by default, or an agency locks things down before a launch and never unlocks them. Nobody notices, because nothing breaks. Traffic from AI answers is hard to see even when it's working.

The trap that reads as "everything is fine"

Here's the shape of the problem. Look at this robots.txt:

User-agent: *
Disallow: /admin

User-agent: GPTBot
Disallow: /

If you (or your tool) only read the User-agent: group — which is what most crawlers and, embarrassingly, our own crawler used to do — this file looks permissive. Nothing blocked except an admin path.

But GPTBot has its own group, and that group closes the whole site. The most specific matching group wins, so never applies to it.

We had to rewrite our robots.txt parser to catch this. It now reads every group and resolves the most specific match, the way a real crawler does. That fix had a nice side effect: our own crawler now obeys rules written for its name, not just *.

How: point it at a log

The evidence you need is already on your server. Access logs record every visit, including which user-agent claimed to be doing the visiting.

So: drop in an nginx or Apache access log, and get an answer.

20002 lines (2 unparsed) — 20000 requests, 5828 from AI crawlers


Googlebot           2375 hits    6 paths  ok=1574 4xx=801  robots=allowed  [search]
GPTBot              1379 hits    6 paths  ok=925  4xx=454  robots=blocked  [training]
ClaudeBot            947 hits    6 paths  ok=636  4xx=311  robots=allowed  [training]
OAI-SearchBot        663 hits    6 paths  ok=453  4xx=210  robots=allowed  [answers]
PerplexityBot        464 hits    6 paths  ok=301  4xx=163  robots=allowed  [search]


absent: ChatGPT-User, Claude-SearchBot, Claude-User, Perplexity-User,
Google-Extended, Applebot-Extended, Applebot, Bingbot, …

(That's a sample log, not a customer's — but it's the tool's real output.)

Suddenly there's something to talk about. Which crawlers came, how often, what they got, and whether the site's own rules let them.

A few design decisions in there that took some thinking:

Not all crawlers are the same thing. Blocking GPTBot costs you nothing in visibility today — it collects training data. Blocking OAI-SearchBot or Perplexity-User costs you presence in answers right now. That's why the report tags each one answers, training or search. "Should we block AI crawlers?" is a licensing and values question for the training ones, and a visibility question for the answering ones. Different conversations.

Some tokens never visit at all. Google-Extended and Applebot-Extended don't exist as crawlers. They're robots.txt controls over how content that Googlebot and Applebot already fetched may be used. A tool that lists them as "never seen" is telling you something meaningless, so we mark them as controls instead.

Spot the contradiction. Look again at that output: GPTBot is blocked — and has 1379 hits. Either the rule is newer than the log, or the crawler ignored it. We only noticed this while reviewing a screenshot of our own output, and it's now flagged explicitly. You can't see it from robots.txt alone.

Your logs stay yours. The file is read where it sits. Never copied into the app, never uploaded. Access logs are full of visitor IP addresses, and this is a local-first tool.

Then stop detecting and start generating

Once the crawlers are coming, the next annoyance: tools that tell you what's missing and leave you to write it.

We flagged missing llms.txt files for a year before it occurred to us that we already had everything needed to write one. A crawl knows the real URLs, the real titles, the real descriptions.

# after a site crawl, in the report:
[ Generate llms.txt ]

# Acme


> We make widgets.


Home



Docs

The tempting shortcut here is to ask a language model to write the file. We deliberately didn't. A generated llms.txt that lists real URLs is verifiable; a hallucinated one is a liability on a client's site. Same with structured data: FAQ schema is built from the question headings actually on the page and the prose beneath them. If a page has no Q&A, you get nothing — and an explanation — rather than invented questions. Schema that contradicts its own page is a manual-action risk, not a quality problem.

Pages that returned an error, have no title, or are marked noindex are left out too. Advertising a page the site itself tells search engines to ignore would be an odd thing for a tool to do on your behalf.

And put it where the work happens

Two smaller things that turned out to matter more than expected.

A CI gate. The analysis engine never needed a window, so now it doesn't have one:

elyra-sja check https://acme.test/pricing --budget 80 --budget-geo 70

Exit codes are the interface: 0 budgets met, 1 a budget missed, 2 couldn't run. Those last two are deliberately different — "the score dropped" and "the site is down" call for different reactions in a pipeline. And budgets work per category, because a GEO regression will happily hide behind a healthy overall score.

Your own assistant can call it. Point Claude, Zed or Cursor at the MCP server and it can analyze pages while you edit content:

{
"mcpServers": {
"elyra-sja": {
"command": "/Applications/Elyra Sjá.app/Contents/MacOS/elyra-sja",
"args": ["mcp"]
}
}
}

Read-only, on purpose. Nothing that writes files or spends an API key is exposed — an assistant in a loop shouldn't be able to edit your checkout through a side door.

The unglamorous conclusion

There's a lot of noise right now about how AI search changes everything, and some of it is true. Answer engines do form descriptions of brands, and being described badly is a real problem that ranking reports won't show you.

But the layer underneath hasn't changed at all. Answer engines retrieve from web indexes. If you're not crawlable, indexed and readable, you're not in the set anything can quote. Retrievability is the precondition, not the previous era.

So before you buy a sentiment dashboard, spend twenty minutes with your access log. You might find the whole problem is one line in a text file.

That's a much better afternoon than it sounds.

Elyra Sjá 0.5.0 is out — AI crawler log analysis, llms.txt and schema generation, a headless CI mode, and an MCP server. It's a single native binary for Apple Silicon, and your data stays on your machine. elyracode.com/sja

Back to blog Raw markdown