A preset replaces the groups below; your sitemaps stay.
*
An empty Disallow blocks nothing: these crawlers may crawl everything.
Optional: seconds between requests. Google ignores it; Bing follows it.
No sitemap yet. Adding its full URL helps crawlers find every page.
User-agent: * Disallow:
robots.txt controls crawling, not indexing: a blocked URL can still appear in Google if other pages link to it. To keep a page out of search results, let it be crawled and add a noindex tag.
Everything runs in your browser: nothing you type is sent to Tooly.
Embed this tool on your site
Free and without an account: copy the code and paste it into your website, blog or docs.
About the Robots.txt & .htaccess Generator
Write a robots.txt without memorizing its syntax. Start from a preset, such as blocking the crawlers that collect pages for AI models, add rules for each user agent and your sitemap, and the file updates as you edit. Then test any URL against it, the way Google reads it. A second tab builds the .htaccess rules most Apache sites need. Everything runs in your browser.
How to use the Robots.txt & .htaccess Generator
-
Start from a preset
Choose Allow everything, Block everything, Block AI crawlers or WordPress, or Blank to start empty. A preset replaces the groups and keeps the sitemaps you added.
-
Edit the groups and test
Add user agents and Allow or Disallow rules to each group, and your sitemap URL. Then type a path or URL in Test a URL and pick a crawler to see whether it is allowed and which rule decided it.
-
Copy or download
Copy the robots.txt or download it and upload it to the root of your domain, so it answers at example.com/robots.txt. For Apache, open the .htaccess tab, tick the rules you need and download that file too.
When to use it
Opting out of AI crawlers
Block GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI crawlers in one click while Googlebot and Bingbot keep crawling the site for search.
Finding why a page is not crawled
Paste a site’s robots.txt into the tester and check a URL with Googlebot to see whether a Disallow rule blocks it, and which one.
Moving a site on Apache hosting
Generate 301 redirects from old URLs to new ones, force HTTPS and choose www or non-www in a single .htaccess file for shared hosting.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing: a blocked URL can still appear in results, without a description, if other pages link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag or X-Robots-Tag header, or remove the page so it answers 404 or 410.
Which crawlers does Block AI crawlers include?
GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, CCBot, PerplexityBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent and cohere-ai, with every other crawler still allowed. Google-Extended and Applebot-Extended only control whether Google and Apple use your pages for their AI models; Googlebot and Applebot keep crawling for search. Blocking OAI-SearchBot also keeps your pages out of ChatGPT search answers. Obeying robots.txt is voluntary, so it cannot stop a crawler that ignores it.
How does the tester decide whether a URL is allowed?
As Google does: it takes the group that names the crawler, or the group for * if none does, and then the rule with the longest matching path. When an Allow and a Disallow match with the same length, Allow wins. In paths, * matches any characters and $ marks the end of the URL. Other crawlers may read unusual cases differently.
Does Google follow Crawl-delay?
No. Googlebot ignores Crawl-delay and adapts its crawl rate to how quickly your server responds. Bing follows it, as the number of seconds between requests. Leave it empty unless a crawler that honors it is overloading your server.
Will the .htaccess file work on my hosting?
Only on Apache, and on LiteSpeed, which reads the same file. Nginx, IIS, Cloudflare Pages, Netlify and Vercel ignore .htaccess and set redirects elsewhere. Each block that needs an Apache module is wrapped in <IfModule>, so a missing module skips the block instead of breaking the site, but keep a copy of your current file before replacing it.
Related tools
Browse all free online tools on Tooly — every one runs in your browser, with no account required.