llms.txt and the Link Plan

Two things the crawl makes possible, both generated rather than written: a file that tells answer engines what your site contains, and a plan for the internal links you are missing — including an honest list of the pages where none belongs.

The problem

An answer engine arriving at your site has to work out what is here and which pages are worth reading. It does this by following links and guessing from titles. Both of those you control, and most sites control them badly: a navigation written for humans who already know what they want, and internal links that exist wherever somebody happened to think of one.

llms.txt, generated from the crawl

llms.txt is a simple convention: a Markdown file at your root listing what your site contains, with real titles and one-line descriptions, so a model does not have to reconstruct your information architecture from scratch.

Sjá generates it from the pages it actually crawled. That phrasing is the entire point, and it has three consequences:

• Real URLs. Every link in the file was fetched successfully during the crawl. A manifest full of 404s is worse than no manifest — it is a machine-readable list of your broken promises.
• Real titles and descriptions from the pages themselves, not invented summaries of pages the generator never read.
• Exclusions that follow your own instructions. Error pages, untitled pages and anything marked noindex are left out. Listing a noindex page in your own manifest is contradicting yourself in public, and a contradiction is a reason to distrust the whole file.

Review it before publishing. It is generated, not sacred, and the ordering is worth a minute of your judgement. Publish at /llms.txt.

The internal link plan

Internal links do two jobs. For search engines they distribute authority and establish which pages are important. For answer engines they define the neighbourhood: a page about pricing that links to the page about the file format is telling a retriever that these two belong to the same subject.

Most sites are under-linked, and the reason is that finding the missing links is dull, repetitive work: for every page, read it, remember the other two hundred pages, and notice where one mentions another without linking. A human does that badly after twenty pages. A large language model can do it, but people price it as a writing task — send the whole page, ask for suggestions, pay for generated prose — and it becomes expensive enough that nobody runs it twice.

It is a classification problem, not a writing one. For every candidate pair the questions are small, closed and repetitive: does this page have a real reason to link to that one, and is there already a phrase in its copy that could carry the link?

So Sjá splits the work. Code finds the candidates — excluding error pages, noindex pages and everything already linked, ranking by the title words the source page's text actually uses, minus site-wide words that would match everything — and finds the anchor phrase. Jev, a judgement model that answers typed questions with calibrated probabilities, answers the two yes/no questions for each pair, in one request per page. A hundred pages costs a few tenths of a cent.

/pricing
  -> /docs/file-format     0.94 / 0.91   "plain Markdown files"
  -> /docs/sync            0.88 / 0.79   "sync across devices"

/blog/why-plain-text
  -> /docs/file-format     0.96 / 0.93   "a folder of .md files"

/about
  nothing fits

That last line is the one to trust the tool for. A plan that finds a link for every page is not a plan you can trust — it is a generator meeting a quota, and following it produces the footer-farm of forced links that has been an SEO smell for fifteen years. Pages where nothing honestly fits are listed as such, and that list is the evidence that the other recommendations were judgements rather than output.

The plan copies as Markdown and writes nothing to your site. You place the links, in your own words, in your own files — which is also where you will notice that two of the twelve suggestions are wrong for reasons only you know.

What you learned

• Generate llms.txt from a crawl, so every URL in it is one that answered.
• noindex pages stay out of your own manifest. Contradicting yourself in public is a reason to be distrusted.
• Internal linking is classification, not writing — which is why it can cost tenths of a cent instead of dollars.
• Code finds the candidates and the phrase; the model only judges. Splitting the work is what keeps it cheap and checkable.
• 'Nothing fits' is the most valuable line in the plan. A tool that always finds something is meeting a quota.
• Nothing is written to your site. You place the links, and you catch the two that are wrong.