How we calculate the robots.txt & Sitemap Validator
What it measures
Whether a site's robots.txt and sitemap.xml are reachable and syntactically valid.
How it works
- 1The tool fetches the live robots.txt and sitemap.xml (or sitemap index) directly from the domain.
- 2Syntax is checked against the documented conventions for each format — valid directive syntax for robots.txt, valid XML structure for sitemaps.
Real limitations
- Validity doesn't guarantee an optimal configuration — a robots.txt can be syntactically valid while still blocking pages you want indexed.
- This checks the files as published; it doesn't crawl every URL listed in a sitemap to confirm each one is actually live.