robots.txt Generator
Generate robots.txt file
π‘οΈ Crawl-delay is non-standard and ignored by some crawlers including Google. robots.txt is not access control.
βΉοΈ robots.txt Generator
Generate robots.txt file
Verify results with an authoritative source before using them for important medical, legal, or financial decisions.