Chapter 8 of 14

The Whole Site

Four chapters of single-page work have taught you what good looks like. Now the same rigour across every page at once — and a class of problem that only exists at site scale, which no amount of looking at one page will ever reveal.

The problem

Some faults are properties of a set of pages, not of any page in it. Two pages with the same title are each perfectly fine on their own; together they are a duplicate that makes a search engine pick one and discard the other, and it may not pick the one you wanted. Your organisation name spelled three ways is not visible from any single page, because each page spells it consistently — with itself.

You cannot find these by auditing pages one at a time, no matter how carefully. You have to hold the whole site in view at once.

The crawl

Seed a URL and a page limit. Sjá discovers pages from robots.txt sitemaps and internal links, respects disallow rules, and waits politely between requests — the delay exists so that auditing your own site never looks like an attack on it, which matters when the site is behind a rate limiter or belongs to a client.

The site crawl view: an overall score of 81 and grade B for wireable.io, with pass, warn and fail counts, per-category scores, llms.txt generation, internal link planning, and an issue inventory.

Start small. Twenty-five pages will find nearly every kind of problem you have, because problems come from templates and templates repeat. A full crawl of four hundred pages mostly re-proves what the first twenty-five told you, at sixteen times the wait.

You can stop a crawl mid-flight and everything crawled so far is kept — badged stopped early, so a partial crawl is never mistaken for a complete one. That badge is a small thing that prevents a large embarrassment: a client report whose “site score” was computed from the eleven pages that happened to finish.

What the site score hides

The site score is an average, and an average is a summary that conceals its own distribution. Freddy's crawl comes back 81, grade B, which sounds like a site needing a bit of polish.

The distribution says otherwise: the docs pages score in the nineties (one template, done well), and the four marketing pages score in the fifties. Those four are the pages anyone actually lands on. An 81 that is really “93 × 20 and 54 × 4” deserves the opposite of a relaxed response.

So read the score, then immediately read the list of lowest-scoring pages, and weight by what each page is for. Ten identical docs pages sharing one flaw is one fix. Your pricing page scoring 54 is one fix too, and it is worth more than the ten.

The issue inventory

The crawl aggregates every finding into an inventory with counts: 15 distinct issues, this one on 10 pages, that one on 8. This is the view that turns an audit into a plan, because a count is a strong hint about where the fix lives.

  • An issue on nearly every page is a template fix. One edit, twenty pages improved. Do these first; they have the best ratio in the whole discipline.
  • An issue on one page is a content fix. Worth doing if the page is worth something.
  • An issue on some pages of a template is the interesting one. It means a conditional: the description is missing only where the excerpt is empty, which is a bug in the template rather than in any page.

The findings only a crawl can make

Broken pages are found and not scored. A 404 is reported as a broken page rather than given a very low score — because averaging a 404 into your site score would drag the number down in a way that says nothing about the quality of your real pages. A dead link is a link problem, not a content problem, and mixing them produces a number that means neither.

Duplicate titles and descriptions across pages. Freddy has five: the docs template fell back to the site name whenever a page had no front-matter title.

The entity spellings. Chapter 7 told you to spell your organisation name identically everywhere. Here is where you find out you have not: Wireable on the landing page, Wireable AS in the footer JSON-LD, wireable.io in the docs publisher block. Three names, one company, and an engine that must decide whether they are the same thing. Pick one and make the others match.

What you learned

  • Some faults belong to the set, not the page. Duplicates and inconsistent entities are invisible one page at a time.
  • Start with twenty-five pages. Problems come from templates, and templates repeat.
  • A stopped crawl is badged, so a partial result is never reported as a whole one.
  • Read the distribution, not the average, and weight pages by what they are for.
  • The issue count tells you where the fix lives — every page means template, some pages means conditional, one page means content.
  • Broken pages are reported, not scored, because a dead link is a different kind of problem.
Next: in Chapter 9 you use the crawl for two things it uniquely enables: an llms.txt built from pages that really exist, and an internal link plan that admits where no link belongs.