Web Crawling Tool

The Web Crawling Tool provides automated, policy-respecting crawling to inventory a site's public topology for SEO audits, content discovery, and site health monitoring. Target users include SEO analysts, webmasters, content managers, and growth teams who need visibility into crawlability, link structure, and content gaps. Use it before site migrations, during large-domain audits, or when benchmarking crawl health against competitors. Core Logic & Features: Required features include seed_url input, max_depth, max_pages, and a robots.txt respect toggle with per-domain politeness. It performs link extraction, metadata capture, and outputs a structured URL set plus a sitemap. Optional/advanced features include JavaScript rendering, credentials for restricted areas, delta/incremental crawls, custom extraction rules, and API access. The domain logic enforces URL normalization, deduplication, error handling, and supports exports in JSON, CSV, or Sitemap XML. Inputs & Outputs: Inputs: start_urls (list of URLs, samples: ["https://example.com"]), max_depth (integer, 0-6), max_pages (integer), obey_robots (boolean), user_agent (string), include_patterns (list), exclude_patterns (list). Outputs: crawled_pages (list of {url, status_code, content_type, response_time_ms, last_modified}), sitemap_xml (string), crawl_summary (object with total_pages, unique_domains, blocked_urls, orphan_pages). Validation enforces valid URLs, non-negative depth, positive page limits, and consistent patterns. Algorithms & Calculations: Crawling uses breadth-first traversal up to max_depth, with a visit queue and a seen URL set to avoid duplicates. It applies robots.txt rules, enforces per-domain crawl_delay, and handles redirects and errors. URL normalization, canonicalization, and per-page scoring are used to support downstream analytics. Outputs include a consolidated crawl_score and blockers metric. Error & Edge Cases: Handles invalid inputs (malformed URLs, negative depth), missing start URLs, DNS failures, timeouts, and rate-limit blocks. If a host blocks crawling, the tool records the event and skips further requests per policy. It gracefully handles circular links and large archives. Industry/Region & Localization: Follows the Robots Exclusion Protocol and ISO timestamps; supports locale-aware URL handling and content encoding. Works with standard SEO data formats and adheres to data privacy considerations when handling public data. Assumptions & Exclusions: Assumes publicly accessible pages; excludes login-protected content by default and does not perform credentialed crawling unless explicitly enabled. JavaScript rendering and API login are optional; bypassing security or terms of service is out of scope.