Skip to content
SkySiteAudit

Crawling a whole website

How to crawl a whole website with a free account: set page limits, depth and exclusions, follow progress live and read crawl-wide findings.

Full-site crawls need a free account and a project.

  1. In the dashboard, add a project for your website.
  2. Open Site Crawler, adjust the page limit, depth and excluded patterns, and start the crawl.
  3. Progress is shown live. You can cancel at any time.

Discovery. The crawler starts from your homepage and the URLs in your sitemaps, then follows internal links found in the HTML. JavaScript-only links are not followed.

Politeness. SkySiteAuditBot obeys robots.txt and Crawl-delay, and makes at most 2 requests at a time to your site.

Limits. Unverified projects can crawl up to 50 pages. After verifying ownership you can crawl up to the published fair-use limit and schedule monitoring.

Crawl-wide checks. Duplicate titles/descriptions/content, trailing-slash duplicates, canonical targets that redirect or fail, and orphan pages (only when the whole site was crawled).