Generate a robots.txt with rules, crawl-delay and sitemap.Generate a robots.txt with rules, crawl-delay and sitemap.