Configure Robots.txt
0 = no delay (recommended for most sites)
Generated robots.txt
Important Notes
- Place robots.txt in your root directory (e.g., example.com/robots.txt)
- Robots.txt is a request, not a security measure
- Always include your sitemap URL
- Don't block important resources (CSS, JS, images)