# Crawlers Setup User-agent: * Crawl-delay: 10 # Paths (clean URLs) Disallow: /blog/tag/ Disallow: /catalog/product_compare/ Disallow: /checkout/ Disallow: /contacts/ Disallow: /index.php/ Disallow: /lens/paramwrapper/ Disallow: /postoptics/ Disallow: /site-map/ Disallow: /virtual/* Disallow: /404/ Disallow: //catalogsearch/ Disallow: /catalogsearch/ # Files Disallow: /xd_receiver.htm Disallow: /*.php$