The robots.txt file instructs search engine crawlers which parts of a site should be crawled and which should be ignored, helping to control crawler behaviour.
How do I block admin or cart pages from being crawled?
Enter your site URL and add a `Disallow: /admin` directive (or the relevant path) in the "Rules" field — the generator builds a valid robots.txt with these rules and a sitemap link.
Does robots.txt remove pages already indexed by Google?
No, robots.txt only tells crawlers not to crawl certain sections — if a page is already indexed, use noindex or the removal tool in Google Search Console to remove it from search results.
Do I need to include a sitemap.xml link in robots.txt?
Not required, but recommended — the `Sitemap: https://example.com/sitemap.xml` directive helps search crawlers find your full list of pages faster.
Common robots.txt directives
| Directive | What it does | Example |
|---|---|---|
| User-agent | Which crawler the rules apply to | User-agent: * |
| Disallow | Block crawling of a section | Disallow: /admin |
| Allow | Allow crawling within a blocked section | Allow: /admin/public |
| Sitemap | Point to the sitemap location | Sitemap: https://example.com/sitemap.xml |