Suggest URLs or paths that Internet search engines should not crawl using the Robots Exclusion Protocol. Search Engines also ...

Suggest URLs or paths that Internet search engines should not crawl using the Robots Exclusion Protocol. Search Engines also use these instructions to find a website's sitemap.  The location of the sitemap will be automatically appended to the bottom of robots.txt.