In order for a website to be found by other people, search engine crawlers, also sometimes referred to as bots or spiders, will crawl a website looking for updated text and links to update their search indexes KB
The robots.txt file needs to be at the root of a site. (Ex: domain.com/robots.txt)
| User-agent: * Crawl-delay: 30 |
Set a crawl delay for all search engines |
| User-agent: * Disallow: |
Allow all search engines to crawl website |
| User-agent: * Disallow: / |
Disallow all search engines from crawling website |