Free Robots.txt Generator
Control which search engines and AI crawlers can access your website. Generate and download your robots.txt instantly.
Understanding robots.txt for WordPress
What is robots.txt?
The robots.txt file is a plain text file placed in your website’s root directory that tells search engine crawlers which parts of your site they should and should not access. It is not a security mechanism - it is a courtesy protocol. Well-behaved crawlers like Googlebot respect the rules in robots.txt, but malicious bots may ignore them entirely. Despite this, a properly configured robots.txt is an important part of your SEO and crawl budget management strategy.
Why block wp-admin and wp-includes?
WordPress stores its admin panel in the wp-admin directory and core system files in wp-includes. Neither of these directories contains content that should appear in search results. Allowing crawlers to index these paths wastes your crawl budget, which is the number of pages Google will crawl on your site in a given period. For small sites this is rarely a problem, but blocking these directories is a best practice that keeps your crawl budget focused on the pages that actually matter: your content, products, and services.
Blocking AI crawlers
AI companies use web crawlers to collect training data from public websites. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google AI), CCBot (Common Crawl), FacebookBot (Meta AI), and Amazonbot (Amazon Alexa) all respect robots.txt directives. If you do not want your content used to train AI models, blocking these crawlers in your robots.txt is the standard approach. This does not prevent your content from appearing in traditional search results, which are handled by different crawlers like Googlebot and Bingbot.
The sitemap directive
Including a Sitemap line in your robots.txt tells search engines where to find your XML sitemap. This is not the only way to submit a sitemap (you can also do it through Google Search Console), but it is the most universal method. Every crawler that reads your robots.txt will discover your sitemap automatically. WordPress SEO plugins like Yoast and Rank Math generate sitemaps at /sitemap_index.xml or /sitemap.xml by default.
Common robots.txt mistakes
The most dangerous mistake is accidentally blocking your entire site with Disallow: /. This prevents all search engines from indexing any page, which can remove your site from Google entirely. Other common errors include blocking CSS and JavaScript files (which prevents Google from rendering your pages correctly), forgetting to include a sitemap reference, and using overly broad Disallow rules that block legitimate content. Always test your robots.txt after making changes using Google’s robots.txt Tester in Search Console.