User-agent: Crawl-delay: 5 Request-rate: 1/10 # maximum rate is one page every 10 seconds