Allow crawling and declare a sitemap
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
This pattern keeps the public site crawlable while giving search engines a sitemap location.
Live SEO tool
Robots.txt Tester
Use a free robots.txt tester to check Allow and Disallow rules, user-agent matching, sitemap directives, and crawl access before publishing.
Robots.txt tells crawlers which paths they can request before they crawl a site. Use this browser-based robots.txt checker to test pasted directives against a crawler name and URL path, review matched rules, and catch risky crawl-blocking mistakes before launch.
CategoryTechnical SEO
Robots input
Paste robots.txt rules, then test a user agent against a path.
Rule result
Review whether the most specific pasted rule allows or blocks the path.
Paste robots.txt rules to test a path.
How To Use
- 1Paste your robots.txt file or draft crawl directives.
- 2Enter the user agent you want to test, such as Googlebot or *.
- 3Enter a URL path such as /admin/, /blog/post, or /assets/app.js.
- 4Review whether the path is allowed or disallowed and which rule matched.
- 5Check sitemap directives and risky broad blocks before publishing.
Best Practices
- 1Use the most specific rule when Allow and Disallow patterns overlap.
- 2Avoid blocking important CSS, JavaScript, images, product pages, articles, and canonical content.
- 3Include a Sitemap directive when it helps crawlers discover important URLs.
- 4Do not use robots.txt as the only way to keep private content out of search results.
- 5Test before launch because small robots.txt changes can affect large parts of a site.
Tool Schema
This template outputs SoftwareApplication schema, FAQ schema, and breadcrumb schema for every tool page that uses the engine.
Examples
Compare stronger and weaker drafts.
Blocking the whole site
User-agent: * Disallow: /
This blocks crawlers from requesting every path and can remove important pages from discovery.
Block private areas only
User-agent: * Disallow: /admin/ Disallow: /internal/
This is safer than broad blocking because public content remains crawlable.
Allow a specific file inside a blocked folder
User-agent: * Disallow: /assets/ Allow: /assets/public-logo.png
When rules overlap, test the exact path so the intended Allow or Disallow behavior is clear.
FAQ
Common questions.
Does this tool fetch my live robots.txt file?
No. It only evaluates the robots.txt text you paste into the browser.
Is this a full crawler simulation?
No. It covers common allow and disallow matching for planning, but production checks should still be verified with search engine tools.
How do I test a robots.txt file before launch?
Paste the final robots.txt draft, choose the crawler you care about, then test representative public pages, blocked folders, rendering assets, and sitemap directives before deployment.
What is a robots.txt tester?
A robots.txt tester checks whether a pasted robots.txt rule set would allow or block a specific crawler and URL path.
Can robots.txt block Google?
Yes. A Disallow rule can prevent Googlebot from crawling matching paths, so broad rules should be tested carefully before launch.
What is the difference between Allow and Disallow?
Disallow tells crawlers not to request matching paths. Allow can reopen a more specific path inside a broader blocked section.
Should robots.txt include a sitemap?
It often helps to include a Sitemap directive so crawlers can discover the canonical sitemap location quickly.
What should a robots txt test tool show?
A useful test should show whether the path is allowed or blocked, which rule matched, which user-agent group applied, and whether broad blocks or missing sitemap directives need review.
Can robots.txt prevent indexing?
Not reliably by itself. Robots.txt controls crawling, not indexing. A blocked URL can still appear in search if discovered through links, so use noindex or access control when removal from search matters.
Related tools
Continue the workflow.
Related articles