Robots.txt Tester

The tool validates robots.txt content and simulates crawler access for specified user agents. It accepts a robots.txt text block or a URL to fetch the file and supports testing against multiple target paths. The output presents per-user-agent results indicating whether each path is allowed or disallowed.
Core logic implements the Robots Exclusion Protocol: user-agent groups, Allow and Disallow rules, and optional Crawl-delay and Sitemap directives. The evaluation adopts the longest-match principle within the relevant user-agent group and supports simple prefix and wildcard matching consistent with common crawlers.
Inputs include the source (text or URL), one or more user-agent identifiers, and test paths. Outputs include a structured results list, a per-path decision, a summary of violations, and a reproducible report suitable for developer review and change tracking.
Edge cases are handled deterministically: missing User-agent groups, unmatched user-agents, conflicting directives, syntax errors, and fetch failures. The tool reports clear error indicators and partial results when partial data is available, without exposing UI concerns. Timeouts and large robots.txt are gracefully managed.
Intended audience includes SEOs, webmasters, and developers preparing site migrations, hosting changes, or policy updates. The tool provides insights aligned with Google Search Console expectations and helps reduce indexing risk by making crawl policies explicit, testable, and auditable.