diagnose-seo

Diagnose SEO

Structured diagnostic framework for crawl issues, canonicalization errors, indexation problems, and rendering failures.

Diagnostic Approach

Technical SEO problems fall into four categories. Diagnose in this order — each layer depends on the previous one working correctly:

1. Crawlability — Can search engines find and access the pages?
2. Indexability — Are the pages allowed to be indexed?
3. Renderability — Can search engines see the full content?
4. Signals — Are the right signals (titles, structured data, links) in place?

Layer 1: Crawlability

Check these in order:

robots.txt

• Fetch [domain]/robots.txt and review the rules
• Look for overly broad Disallow rules blocking important paths
• Verify Sitemap: directive points to the correct sitemap URL
• Check for different rules per user-agent (Googlebot vs others)

Common mistakes:

• Disallow: / blocking the entire site (often left from staging)
• Blocking CSS/JS files that Googlebot needs for rendering
• Blocking API or AJAX endpoints that load dynamic content
• Staging robots.txt accidentally deployed to production

XML Sitemap

• Fetch the sitemap URL(s) and check:
  • Does it return 200? Is it valid XML?
  • Does it list all important pages?
  • Does it exclude pages that shouldn't be indexed (404s, redirects, noindex pages)?
  • Are <lastmod> dates accurate and recent?
  • For large sites: is there a sitemap index?

Site Architecture

• Pages should be reachable within 3 clicks from the homepage
• Check for orphan pages (no internal links pointing to them)
• Check for redirect chains (page A → B → C — should be A → C)
• Check for redirect loops

Server Response

• Do all important pages return HTTP 200?
• Check for unexpected 301/302 redirects
• Check for soft 404s (page returns 200 but shows "not found" content)
• Verify HTTPS is enforced (HTTP should 301 to HTTPS)

Layer 2: Indexability

Meta Robots / X-Robots-Tag

• Check for <meta name="robots" content="noindex"> on pages that should be indexed
• Check HTTP headers for X-Robots-Tag: noindex
• Common cause: CMS accidentally applying noindex to pagination, tag pages, or new pages