What you’ll need
Paste a robots.txt file and paths, or fetch a public site.
SEO TOOL · FREE TO USE
robots.txt Tester
Test paths against robots.txt and see the matching rule.
Free to use · No saved inputs01Crawling workspace
- 1 Add your input
- 2 Run the tool
- 3 Get your results
Paste a robots.txt file and paths, or fetch a public site. New here? Start with “Try an example”.
More settings (optional)
Local processing · No sign-up
How it works
RFC-style crawler grouping, longest matching rule and allow-on-tie. Live robots failures are reported separately.
Know the limits
Rules describe crawler permission, not indexing or whether a crawler actually obeys them.
GET FROM A QUESTION TO A NEXT STEP
Find the rule blocking your page
An important page is missing from search, or a new robots.txt rule might block more than you intended. Start by checking the exact path against the intended crawler. This robots.txt tester shows the rule behind its answer so you can change a specific directive rather than removing every restriction. It is useful before a launch, migration or crawler-policy update.
How to use robots.txt Tester
Paste the robots.txt rules you want to check, or choose a live check for a public website. Keep a copy of the current file before editing it.
Add the paths you care about and select the crawler. Include both a page that should be accessible and one that should remain restricted.
Run the test and inspect the matching rule for each path. After making a change, test the same paths again before publishing the file.
What do the results mean?
Allowed means the supplied robots rules permit crawling for that agent and path. It does not mean the page is indexed, healthy or accessible through every security layer. A failed live request is different from a confirmed disallow rule; investigate unknown results before changing policy.
What to do next
If permission looks correct but content is still missing, inspect the page response, canonical and noindex directives with Fetch & Render.
Keep the scope in mind
Rules describe crawler permission, not indexing or whether a crawler actually obeys them.
Common questions about robots.txt Tester
Does blocking a URL remove it from Google?
Do not use robots.txt as a reliable removal method. A crawler may still discover a blocked URL. Indexing controls and access protection address different problems; check Google’s documentation before combining them.
Why does a more specific rule change the answer?
Several rules can match a path. This tester evaluates the selected agent group and matching specificity, then shows the decisive directive. Test query strings and trailing slashes exactly as they appear in your URLs.