Puffinmint
EN
Italiano English Español Français Deutsch Português (Brasil) 日本語 한국어 Polski Türkçe

Robots.txt tester: check URLs and user agents

Test robots.txt rules for a URL and user agent. See the matching allow or disallow rule; crawl permission alone does not guarantee search indexing.

Settings

Enter a path such as /page or a domain with a path, with or without HTTPS. The test only reads the pasted robots.txt.

The tester applies product-token matching, merged groups, the most specific rule and Allow on a tie. It does not simulate proprietary extensions. Robots.txt is not access control and does not guarantee removal from search. Maximum 500 KiB.

Your input is sent to the server for inspection. No authentication is sent to the destination site.

Result

We count which tools are used most, without including entered values or file contents in the statistics. Privacy

How to use the tool

Paste robots.txt, enter a path and the crawler product token, such as Googlebot. The text is evaluated without fetching the site.

How the check works

The tester compares the path against rules for the applicable product-token group. It merges equivalent groups and selects the most specific rule; Allow wins a tie.

Interpreting the result

In the example /private/report is blocked, while /private/public/report is permitted by the more specific rule. Crawl permission does not guarantee indexing or protect private data.

Illustrative example

Illustrative example with sample data: actual responses may change.

Input
User-agent: * Disallow: /private/ Allow: /private/public/
Result
/private/report → Disallow · /private/public/report → Allow

In the example /private/report is blocked, while /private/public/report is permitted by the more specific rule. Crawl permission does not guarantee indexing or protect private data.

Limits and data handling

The tester applies product-token matching, merged groups, the most specific rule and Allow on a tie. It does not simulate proprietary extensions. Robots.txt is not access control and does not guarantee removal from search. Maximum 500 KiB.

Your input is sent to the server for inspection. No authentication is sent to the destination site.

Results reflect the time of the check.

Sources

Frequently asked questions

What are the limits of this check?

The tester applies product-token matching, merged groups, the most specific rule and Allow on a tie. It does not simulate proprietary extensions. Robots.txt is not access control and does not guarantee removal from search. Maximum 500 KiB. Results reflect the time of the check.

You might also need

View all

DNS lookup: check a domain’s DNS records

Look up a domain’s A, AAAA, MX, TXT, NS, CNAME and CAA records. Inspect responses and errors without treating one resolver as a global propagation test.

WHOIS lookup: public domain data with RDAP

Look up a domain’s published registrar, dates, status and nameservers with RDAP. Redacted or unavailable registration fields remain clearly identified.

Security headers check: inspect HTTP responses

Inspect a page’s CSP, HSTS and other HTTP response headers. Read actual values and contextual explanations; this check does not certify website security.

Favorites