Skip to content
SkySiteAudit

robots.txt and sitemap validator

How SkySiteAudit parses robots.txt and XML sitemaps, what each warning means, and which robots.txt problems can stop search engines crawling.
  • robots.txt missing (404) — allowed; crawlers treat everything as crawlable. We show a warning because most sites benefit from referencing a sitemap.
  • robots.txt error (5xx or timeout) — serious: search engines may stop crawling the whole site until it recovers.
  • Unrecognised lines — lines that are not User-agent, Allow, Disallow, Sitemap, Crawl-delay or other known directives. They are ignored by crawlers.
  • Path test — evaluates a path for Googlebot using standard longest-match rules.
  • Sitemaps — we read up to 10 sitemap files and 5,000 URLs, including sitemap indexes and gzip files, and flag invalid or cross-host URLs.