Build a robots.txt file to control how search engines crawl your site. Use common presets or create custom allow and disallow rules.
Get notified about new SEO tools
More free tools coming soon — keyword research, sitemap generator, and more.
Robots.txt is a text file at the root of your website (example.com/robots.txt) that tells search engine crawlers which pages they can and cannot access. It is the first file crawlers check before indexing your site. While it is not a security measure — crawlers can still access blocked URLs — it helps manage your crawl budget and keep private areas out of search results.
Yes. You should disallow crawling of admin areas (/wp-admin/, /admin/), login pages, staging environments, and any pages that don't provide value in search results. This prevents search engines from wasting crawl budget on non-public pages.
Place the robots.txt file in the root directory of your website so it is accessible at yourdomain.com/robots.txt. Most web hosting providers and CMS platforms have a designated place for this file. For WordPress, you can use a plugin or place it directly in your public_html folder.
Get notified about new SEO tools
More free tools coming soon — keyword research, sitemap generator, and more.