Robots.txt Tester: Check if a URL Is Blocked

How to use

1. Paste a robots.txt file, or enter a site address and load its live file.
2. Choose a crawler (user agent) and enter one or more paths or URLs, one per line.
3. Each line shows Allowed or Blocked and the rule that matched.

What this tool does

The tester follows the Robots Exclusion Protocol (RFC 9309), the rules Google and Bing use: the crawler obeys the most specific matching user-agent group, the longest matching Allow or Disallow rule wins, a tie goes to Allow, * matches any characters and $ anchors the end of the URL. It also lists syntax problems such as rules outside a group and unknown directives.

Why it matters

One stray Disallow line can stop search engines from crawling a whole site. Testing important URLs before you change robots.txt prevents expensive mistakes.

Common uses

• Checking that CSS, JavaScript and images used by pages are not blocked.
• Testing a new robots.txt before uploading it.

Tips

• robots.txt controls crawling, not indexing. Use noindex to keep a page out of search results.
• Paths are case-sensitive: /Blog/ and /blog/ are different.

Limitations

• Crawl-delay and non-standard directives are reported but not simulated.
• Pasted files are tested in your browser. Loading a live file asks our server to fetch https://the-site/robots.txt.