Configure crawling rules for search engine bots and generate a properly formatted robots.txt file.
Configuration
User Agent
Crawl Delay (optional)
Disallow Paths
Allow Paths
Sitemap URL
Generated Output
Your robots.txt content will appear here...
Create and validate robots.txt files to control search engine crawling
Configure crawling rules for search engine bots and generate a properly formatted robots.txt file.