Crawl rules
Review user-agent groups, allow rules, disallow paths, crawl-delay values, and invalid directives in one readable summary.
Free Tool
Free Robots.txt Checker
A robots.txt checker fetches a site's robots.txt file and reviews crawler rules, user-agent groups, sitemap directives, and SEO issues that can accidentally block Googlebot or other crawlers.
How to test robots.txt rules
- Enter the website URL. Paste the home page or robots.txt URL for a public site.
- Fetch robots.txt. The checker normalizes the URL to /robots.txt and reads the crawler rules.
- Review crawler groups. Check user-agent groups, allow and disallow rules, crawl-delay, and sitemap lines.
- Fix SEO warnings. Add missing sitemap directives, remove accidental sitewide blocks, and correct invalid directives.
Sitemap checks
Confirm that the file points search engines to XML sitemaps so canonical URLs can be discovered quickly after releases.
SEO warnings
Flag missing files, sitewide blocks, blank user agents, oversized files, and directives crawlers may ignore.
Robots.txt checker FAQ
What is a robots.txt checker?
A robots.txt checker fetches a site's robots.txt file and reviews crawler rules, user-agent groups, sitemap directives, and common SEO issues that affect search engine access.
Does robots.txt block pages from being indexed?
Robots.txt can block crawling, but it is not a guaranteed noindex control. To remove a page from search results, use noindex on a crawlable page or remove the page from the site.
Should every site have a robots.txt file?
Most public sites should have a robots.txt file because it gives crawlers a predictable place to find rules and sitemap URLs, even when the rules are simple.
What does Disallow: / mean?
`Disallow: /` under `User-agent: *` tells most crawlers not to crawl any public path on the site. That is useful for staging sites, but dangerous on production sites.
Is this robots.txt tester safe for private URLs?
The checker only accepts public http and https URLs. It blocks localhost, private network, and reserved address ranges before fetching robots.txt.