Search Infrastructure
That Crawlers Trust

Search infrastructure is the part of the site crawlers depend on before any ranking debate starts. It is not a list of SEO tips. It is the contract between your server and a bot.

A crawler arrives. It fetches HTML. It parses the DOM. It follows links. It reads directives. It decides whether to store the page. Break any step and content quality does not get a vote. The URL is not in the game.

The stack is specific: robots.txt, sitemaps, canonicals, redirects, internal links, semantic HTML, structured data, Core Web Vitals, and whether AI bots are even allowed in. Each piece is a response the crawler can verify. A workshop slide is not a response.

Third-party site-health benchmarks in 2025 put the failure rate at 72% for at least one critical technical factor. That is the default state of the web — not a rare disaster. Most teams find out when traffic drops, not when the tag shipped wrong.

Crawlers do not trust brands. They trust responses: a consistent, parseable HTML document, returned quickly, matching what a person sees.

Trust breaks in three boring ways. The server shows the bot different HTML than the user. robots.txt blocks a crawler nobody meant to block. The copy lives behind JavaScript the bot will not run.

Google’s mobile-first indexer is the primary index. Incomplete or slow mobile HTML is what gets stored. Desktop looking fine is not evidence. WordPress is a useful baseline: Custom Web Audits put Core Web Vitals pass rate at 43.44% — the most common CMS, and more than half fail the field test.

A 200 with clean HTML this week, a 500 next week, and a redirect chain the week after teaches the crawler to visit less. Crawl budget shrinks. New URLs wait. That is not a ranking mystery. That is a trust pattern.

1. 01

   Crawl blockers

   robots.txt mistakes, server errors, accidental noindex. If Google cannot fetch the URL, nothing else on this page matters.

2. 02

   Index blockers

   Canonical conflicts, redirect chains, duplicates. The crawler arrived. The index still said no.

3. 03

   Render blockers

   Content that only exists after JavaScript. Google may get there later. AI crawlers will not.

4. 04

   Performance

   Core Web Vitals on pages that are already in the index. A fast page nobody can find is still a hobby.

Crawling is fetching. Indexing is storing. Rendering is executing. They are three queues. Most teams treat them as one checkbox.

A URL can be crawled and not indexed. Indexed and rendered days later. Rendered and still canonicalized away. If you cannot name which step failed, you cannot pick a fix — you can only ship another plugin.

Google Search Console is the authoritative source for indexation and Core Web Vitals field data. Lighthouse is a diagnostic. It tells you what to try. GSC tells you whether Google agrees it worked.

AI crawlers read raw HTML. They do not execute JavaScript. If the article lives in a client-rendered shell, GPTBot and PerplexityBot see an empty page. SSR or SSG is the floor for AI visibility — not a nice-to-have for “later.”

robots.txt is a gate. A sitemap is a hint. Crawl budget is what is left after both.

Absence of a Disallow means the path is allowed. Explicit Allow is belt-and-suspenders — useful when a broader rule sits higher in the file. The usual failure is inheritance: a CMS default, a plugin, a staging copy that reached production. Nobody reviewed the file. The bot did.

Do not use robots.txt to hide a page you still want indexed. Disallow stops the fetch. noindex needs the fetch. Mixing those two is how teams block URLs they meant to deindex — and deindex URLs they meant to keep crawlable.

Sitemaps do not command indexation. They help discovery. Submit canonical, indexable, 200 URLs. A file of 50,000 rows that includes redirects, 404s, and duplicates teaches Google to trust the sitemap less. Clean it like a promise.

Crawl budget bites on large sites and on small ones that generate infinite parameter URLs. Faceted navigation, session IDs, and crawl traps spend the visit on junk. Important URLs wait. Block the waste or canonicalize it — do not hope the bot “figures it out.”

A canonical tag is a suggestion Google usually follows. A 301 is a command the crawler obeys. Know which one you shipped.

Duplicate URLs are dilution, not a cartoon penalty. Same document at three addresses and Google picks one. Your canonical says which. If the tag, the redirect, and the sitemap name three different winners, Google picks anyway — and it may not be the URL in your analytics.

Redirect chains of three or more hops waste crawl time and delay the destination. Collapse every chain to a single 301. A 301 to a 302 to a 200 is a chain. A 301 to a 301 to a 200 is a chain. One hop.

Migrations fail here more than in the CMS. Old path, new path, leftover www, leftover trailing slash, leftover parameter. Audit canonicals, redirects, and sitemaps as one set. Faceted filters that mint indexable combinations will keep minting duplicates after you “fixed SEO.”

Internal links are how crawlers find pages and how Google reads the shape of the site. The sitemap is backup, not the map.

Keep money pages close to a crawl entry — homepage, hub, or a page that already has authority. Depth-four URLs get fetched less often. That is not a published Google law. It is what you see in logs when everything important sits under five folders and a mega-menu nobody clicks.

Anchor text is a label. “Click here” tells the crawler nothing. Descriptive, consistent anchors beat stuffed replicas of the target title. Orphans — 200s with no in-links — are the quiet indexation failure. They sit in the sitemap. Navigation never votes. Link the ones that matter. Remove the rest.

Location pages, tag archives, and filter combinations are IA problems wearing a content costume. If the template cannot say something unique, do not ask the crawler to treat each URL as a document.

Crawlers do not read the Figma file. They read the HTML. Div soup with no headings is a guess. Semantic structure is a map.

One H1. H2s that match the questions on the page. A title tag that names the document. A meta description that earns the click — it is SERP UI, not a ranking lever. These are the minimum contract, not “on-page tricks.”

JSON-LD is how you say Article, Product, FAQ, Breadcrumb, Organization without making the parser infer it. Useful for rich results and for entity clarity. Invalid schema is worse than none: you claimed a type and failed the test. Validate before you ship, not after GSC enhancements turn red.

Trustworthy content still needs a trustworthy page. E-E-A-T does not rescue a document the crawler cannot parse — or a brand entity with no Organization markup and no author.

Core Web Vitals are field data from real users. Lighthouse is how you debug. GSC is whether you passed.

Thresholds at the 75th percentile: LCP ≤ 2.5s, INP ≤ 200ms, CLS ≤ 0.10. INP replaced FID in March 2024. There is no bonus for a 100 lab score on a URL that fails in the field.

Google has published a 32% higher bounce at a three-second load. Speed is retention, not vanity. It is still the last layer in the fix order. A fast page that is not indexed is worth nothing.

GSC field data rolls on about 28 days. Ship the fix. Do not freeze the roadmap waiting for the chart. Check next month whether the template actually moved.

An audit that does not produce a fix order is a report. Reports describe. Audits decide Monday.

Start in GSC: coverage, sitemaps, Core Web Vitals. Then logs for what actually crawled. Then a full-site crawl for chains, orphans, and canonical fights GSC will not list as a single error. A site audit without GSC is incomplete — the first deliverable is access, not a Lighthouse PDF.

Quarterly is the default cadence. Monthly for large catalogs. Same-day after a migration, replatform, or CDN change. Traffic that falls 48 hours later is not a mystery if nobody crawled the new templates before launch.

Priority does not rotate with the calendar: indexation blockers, then crawl waste, then CWV, then schema, then copy. Fix the gate before you landscape the garden.

If the operator is out next Monday, does the check still happen — or does infrastructure wait for a hero?

1. Are there new errors in GSC coverage since last week?
2. Did indexed count drop without a reason you can name?
3. Are submitted sitemaps returning 200 and processing?
4. Any redirect chain of three or more hops?
5. Orphan URLs in the sitemap with no internal links?
6. Does robots.txt block a path you meant to rank?
7. Are GPTBot, PerplexityBot, and ClaudeBot allowed — in robots.txt and Cloudflare?
8. Do canonical tags disagree with redirects or sitemap URLs?
9. Are CWV failing URLs in GSC going up or down?
10. Is the mobile HTML the same as View Source — and as GSC rendered HTML?

01/

What is the difference between crawling and indexing?

Crawling is the fetch. Indexing is storage. A page can be crawled and still excluded — duplicates, thin content, noindex. GSC coverage tells you which step failed. Guessing from analytics does not.

02/

Does Google execute JavaScript?

Yes, on a separate render queue that can lag days behind the first crawl. If the content only appears after hydration, Google may index an empty shell first. AI crawlers like GPTBot do not execute JavaScript at all. SSR or SSG is the floor.

03/

What are Core Web Vitals thresholds?

Field data, 75th percentile: LCP ≤ 2.5s, INP ≤ 200ms, CLS ≤ 0.10. GSC and CrUX are the scoreboard. Lighthouse is a diagnostic. INP replaced FID in March 2024 — do not still report FID.

04/

Is robots.txt required for SEO?

Not required. Expected. Absence of a Disallow means allowed. Explicit Allow is belt-and-suspenders. The failure is almost never “we forgot the file.” It is a CMS default, a plugin update, or a staging copy that now blocks the URLs you wanted crawled.

05/

Do sitemaps guarantee indexing?

No. A sitemap is a hint, not a command. Submit canonical, indexable, 200 URLs. Mix in redirects, 404s, and duplicates and Google learns to trust the file less.

06/

How do I know if AI crawlers can access my site?

Fetch robots.txt for GPTBot, PerplexityBot, and ClaudeBot. Then check Cloudflare — the “Block AI Bots” default catches teams who never opened the panel. Confirm with logs, not a dashboard toggle you hope is off.