Screaming Frog · SEO crawl & architecture
Your site as a bot sees it: URLs, status codes, metadata.
FREEMIUM
Why I use it
- See the full architecture in one pass, including index bloat.
- Compare two crawls to measure before/after migration or redesign.
How I use it : mini guide
I respect robots.txt, then export canonicals, titles, hreflang when present.
- Inventory : HTTP codes, redirect chains, orphan or overly deep pages.
- Duplicates : Duplicate titles/meta, duplicate content on URL variants ; map fixes in Shopify (redirects, canonicals).
- Hreflang & markets : On multi-market setups, check per-locale URL consistency.
- Exports : CSV shared with the client to batch-fix issues.
I don’t crawl at max depth without excluding preview or faceted URLs when needed : exclusion filters to reduce noise.
Concrete Shopify example
Migration: 302s instead of 301s on part of the catalogue. Before/after crawl comparison after fixing redirects.
Limits I've identified
- 500 URLs on the free tier : beyond that, licence or split crawls.
- Doesn’t replace Shopify business logic (dynamic collections, etc.).
- Heavy CPU on very large sites.
Alternatives if you don't want to pay
- Sitebulb or Oncrawl for more visual crawl maps if budget allows.