Web Indexer

The Seagull Web Indexer is a domain-level tool that builds a language-aware inventory of a site's HTML pages and generates a crawlable sitemap. It targets SEO engineers, content strategists, and web-ops teams who need a verifiable URL catalog, multilingual coverage, and exportable outputs for audits, migrations, or ongoing SEO workflows. Core logic includes URL normalization, language detection, and content-type filtering, then crawls up to a configurable max_depth with optional query-string handling. Advanced features include hreflang generation, last-modified extraction, and dual outputs (JSON index and XML sitemap) optimized for search engines.
Outputs are exportable as JSON, XML, or CSV and can be validated against RFCs for canonical URLs. Robots.txt awareness and exclusion rules help avoid restricted paths. The tool supports UTF-8 and locale-aware metadata to improve international indexing.
Ideal users are SEO analysts, site builders, and marketing engineers performing migrations, multilingual deployments, or ongoing site health checks. The unique value lies in its combination of language-aware indexing, export-ready formats, and strict URL normalization that reduce crawl waste and improve coverage.
Use cases include validating new language variants before launch, auditing legacy pages for redirections or duplicates, and preparing data for search console or webmaster tools.