Robots.txt Generator
Create a well-formed `robots.txt` file to guide search engines on how to crawl your website effectively.
Default Policy for All Crawlers
Set the default access policy for all user-agents (`*`).
Sitemap URL (Recommended)
Include the full URL to your XML sitemap.
Custom Rules for Specific Bots
Add specific `Disallow` rules for individual crawlers.
Generated robots.txt
# Your generated robots.txt file will appear here.Copy the generated text and save it as `robots.txt` in the root directory of your website.
How to Use This Tool
1Set Default Policy
Choose whether to allow or disallow all search engines (`User-agent: *`) by default. "Allow All" is recommended for most websites.
2Add Sitemap URL
Paste the full URL to your `sitemap.xml` file. This is highly recommended as it helps search engines discover all your important pages.
3Add Specific Rules
If you need to block specific bots from certain sections, click "Add New Rule". Select the bot and enter the path you want to disallow (e.g., `/admin/` or `/private-images/`).
4Generate & Copy
The `robots.txt` content is generated automatically on the right. Once you're happy with the rules, click the "Copy" button.
5Upload to Your Site
Create a new file named exactly `robots.txt` (all lowercase). Paste the copied content into this file and upload it to the **root directory** of your website (e.g., `https://www.aeoclub.com/robots.txt`).
Frequently Asked Questions
A `robots.txt` file is a plain text file that tells search engine crawlers (like Googlebot) which pages or files the crawler can or can't request from your site. It acts as a guide, not an unbreakable command, but all reputable bots respect its rules.
The file must be named `robots.txt` and placed in the **root directory** of your website's host. For example, for `www.aeoclub.com`, the file should be accessible at `https://www.aeoclub.com/robots.txt`. It will not work in any other location.
A `User-agent` is the specific name of the crawler or bot you want to give instructions to. `User-agent: *` is a wildcard that applies to all bots. `User-agent: Googlebot` applies only to Google's main crawler. This allows you to set different rules for different search engines.
This is a critical distinction.
• `Disallow: /` (with a slash) tells bots not to crawl **any page** on the entire site. It effectively blocks your whole website.
• `Disallow:` (with nothing after it) means nothing is disallowed, effectively allowing bots to crawl everything. This is the correct rule to use for "Allow All".