Robots.txt generator for Blogger and WordPress
Build a correct robots.txt from presets or your own rules, with the sitemap line included.
Runs entirely in your browser. Nothing you enter is sent anywhere.
Start each path with a forward slash. Ending with a slash blocks the whole folder.
Google ignores this directive. Bing and Yandex honour it.
How to use the robots.txt generator
- Choose a preset that matches your platform. Each one blocks the paths that genuinely need blocking on that system and nothing else.
- Enter your site URL and sitemap URL. The sitemap line is the single most valuable thing in a robots.txt file — it tells every crawler where your URL list lives.
- Add any extra paths to block, one per line. Checkout pages, thank-you pages, internal search results and staging folders are the usual candidates.
- Copy the output or download it, then upload it to your site root so it sits at yoursite.com/robots.txt. It has to be at the root — a robots.txt in a subfolder is ignored entirely.
What you can use it for
Keeping thin and duplicate pages out of the crawl is the main job. Internal search result pages, faceted filter URLs and paginated archives can generate thousands of near-identical URLs, and letting a crawler wander through all of them wastes the crawl budget that should be spent on your actual content.
Protecting staging and development sites is the highest-stakes use. A staging site that gets indexed competes with your live site for the same keywords and is a genuine SEO problem. The block-everything preset exists for exactly this, though HTTP authentication is stronger and should be preferred where you can use it.
Blocking SEO crawlers such as Ahrefs and Semrush stops competitors profiling your backlinks and content. It is a legitimate choice, though it cuts both ways — you lose your own data in those tools too, and it does nothing to hide anything from Google.
AI crawler directives are a newer decision. GPTBot and CCBot respect robots.txt, so blocking them keeps your content out of certain training datasets. It is worth deciding deliberately rather than by default, since some publishers want the visibility that comes with being cited by AI assistants.
Things to know about robots.txt
Robots.txt controls crawling, not indexing. This distinction causes real damage when it is missed. A blocked page can still appear in search results — Google just shows it without a description, because it was told not to look. To keep a page out of the index, allow it to be crawled and use a noindex meta tag on it instead.
Blocking a page also blocks the noindex tag on it. If you disallow a URL in robots.txt and put noindex in its HTML, the crawler never fetches the page and never sees the tag, so the page can stay indexed indefinitely. Choose one mechanism, not both.
The file is a public request, not a security boundary. Anyone can read yoursite.com/robots.txt, and well-behaved crawlers honour it while malicious ones ignore it entirely. Listing /secret-admin/ in your robots.txt tells the world exactly where to look. Never use it to hide anything sensitive.
Google has ignored the crawl-delay directive for years, and uses its own crawl rate calculation instead. Bing and Yandex do honour it. If Googlebot is genuinely overloading your server, the setting to change is in Search Console, not in this file.
Frequently asked questions
At the root of your domain, so it resolves at yoursite.com/robots.txt exactly. It cannot live in a subfolder. On Blogger you do not upload a file at all — go to Settings, Crawlers and indexing, and enable custom robots.txt, then paste the content there.
No, and this is the most common misunderstanding. Blocking stops the crawler reading the page but does not remove it from the index — it may still be listed with no description. To remove a page, let it be crawled and add a noindex meta tag, or use the removals tool in Search Console for something urgent.
Not strictly. A missing robots.txt means everything is crawlable, which is fine for a small site. Having one is still worth it for the sitemap line, and because a 404 on that path appears in every crawl log as a minor error.
It is a genuine trade-off rather than an obvious answer. Blocking keeps your content out of training data. Allowing means your site can be surfaced and cited by AI assistants, which is becoming a real traffic source. Publishers are split, and both positions are defensible.
It blocks the entire site from that user agent. It is correct for a staging environment and catastrophic on a live site — a stray Disallow: / has removed more sites from search than almost any other single mistake. Always check your live robots.txt after a deployment.