Typical site
- Input
- Disallow /admin/ and /cart/, allow everything else
- Output
- User-agent: *\nDisallow: /admin/\nDisallow: /cart/
Private areas are kept out of the index while the rest of the site stays crawlable.
Robots.txt Generator
A robots.txt file sits at the root of your domain and tells crawlers which parts of your site they may request. This generator builds one from simple rules, warns about the mistakes that accidentally block a whole site, and lets you test any path against the result before you publish.
Runs entirely in your browser — nothing you type is uploaded.
One absolute URL per line. The most useful line in the file.
No sitemap listed. A Sitemap: line is the most useful thing robots.txt can carry.
Test a path
Blocked — Matched by Disallow: /admin/ in the * group.
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart/Save this as robots.txt at the root of your domain, reachable at /robots.txt.
How to
How to use the Robots.txt Generator
- 1
Choose which crawlers
Start with a rule for all crawlers, then add groups for specific bots such as Googlebot or GPTBot.
- 2
Add allow and disallow paths
Disallow the paths crawlers should not request, such as /admin/ or /cart/. Paths are matched from the start of the URL.
- 3
Add your sitemap
Point crawlers at your sitemap with its full absolute URL. This is the single most useful line in the file.
- 4
Test and publish
Check a few paths in the tester, then save the file as robots.txt at the root of your domain.
Examples
Robots.txt Generator examples
Blocking the whole site by accident
- Input
- Disallow: /
- Output
- Warning: this blocks every crawler from the entire site
A single slash blocks everything. It is the most common and most damaging robots.txt mistake, so the tool flags it.
Blocking AI crawlers only
- Input
- GPTBot and CCBot disallowed, others allowed
- Output
- User-agent: GPTBot\nDisallow: /
Named groups let you keep search crawlers while excluding AI training bots. Each group applies only to the agent it names.
Why use it
What the Robots.txt Generator gives you
Catches the disaster cases
A bare Disallow: / on all agents deindexes a site. The tool warns before you copy anything that would do that.
Test paths before publishing
Enter any URL path and see which rule matches and whether it is allowed, using the same longest-match logic crawlers use.
Presets for real crawlers
One click adds correctly spelled groups for Googlebot, Bingbot, GPTBot, ClaudeBot and others — misspelled agent names are silently ignored by crawlers.
Correct group syntax
Rules are grouped per user-agent with proper ordering, which is where hand-written files most often go wrong.
Free and private
Everything runs in your browser and no account is needed.
Good to know
Robots.txt Generator limitations
- Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it — use a noindex meta tag to keep a page out of the index.
- It is advisory. Well-behaved crawlers obey it; scrapers and malicious bots ignore it entirely.
- The file must sit at the domain root, at /robots.txt. In a subdirectory it has no effect.
- Blocking a page with robots.txt stops Google reading its noindex tag, so the two must never be combined on the same URL.
Summary
Robots.txt Generator in short
- Robots.txt tells crawlers which paths they may request; it must live at the domain root.
- It controls crawling, not indexing — blocked pages can still be listed if linked elsewhere.
- Disallow: / blocks an entire site and is the most common serious mistake.
- Never block a URL you also want to carry a noindex tag, as the crawler will never read it.
FAQ
Robots.txt Generator questions
What is a robots.txt file?
A plain text file at the root of your domain that tells crawlers which paths they may request. It is the first thing most crawlers fetch when they visit a site.
Where does the generated robots.txt go?
At the root of the domain, reachable at https://yourdomain.com/robots.txt. Crawlers look only there — a file in a subdirectory is ignored completely.
Does blocking a page remove it from Google?
No. Robots.txt stops a page being crawled, not indexed. A blocked URL that other sites link to can still be listed, usually without a description. To remove a page, allow crawling and add a noindex meta tag.
Can I use Disallow and noindex together?
No, and combining them backfires. If robots.txt blocks the URL, the crawler never fetches the page and never sees the noindex tag, so the page can stay indexed indefinitely.
Do wildcards work?
Google and Bing support * to match any sequence and $ to match the end of a URL, so Disallow: /*.pdf$ blocks PDF files. Support is not universal, so keep rules simple where you can.
Is Crawl-delay respected?
Google ignores it entirely — crawl rate is managed in Search Console. Bing and Yandex do honour it. Include it only if those crawlers are overloading your server.
How do I block AI crawlers?
Add a group for each bot, such as GPTBot, ClaudeBot, CCBot or Google-Extended, with Disallow: /. The names must be exact — a typo means the rule silently does nothing.
Which rule wins when two match?
The most specific match wins, meaning the longest matching path, not the first one listed. Allow: /blog/public/ overrides Disallow: /blog/ because it is longer.
What if I do not need to block anything?
Publish a file with User-agent: * and Disallow: left blank, plus your sitemap line. That allows everything explicitly and still gives crawlers the sitemap pointer.
Discover
Related SEO tools
Sitemap GeneratorTurn a list of URLs into a valid XML sitemap with priority, change frequency and last-modified dates. Copy or download the file.Canonical Tag CheckerCheck any page's canonical tag, see whether it is self-referencing, and catch redirect and mismatch problems before Google does.Schema GeneratorGenerate valid JSON-LD schema markup for articles, products, local business, events and more. Copy the code and paste it into your page.Meta Title GeneratorGenerate SEO title tags from your keyword and see exactly where Google truncates each one, measured in pixels rather than characters.