Crawler directives
robots.txt controls crawling behavior but does not always prevent indexing.
Robots.txt Generator
Build robots.txt groups, rules and sitemap directives without crawling your site.
URLs and metadata are processed locally without crawling or external transmission.
0 characters · 0 bytes → 0 / 0
How to use Crawler directives
- 1
Enter or load the data locally.
- 2
Choose the visible options and process the content.
- 3
Review warnings before copying or downloading.
Features and limitations
Local privacy
Files are not uploaded and inputs are not automatically stored.
Safe export
Copy or download UTF-8 results created in the browser.
Frequently asked questions
How does this tool work?
robots.txt controls crawling behavior but does not always prevent indexing.
Is my data uploaded?
No. Processing stays in the browser and uses no external APIs.
What are its limitations?
Validation is local and syntactic; remote resources are not queried or verified.