# Crawlers SetupUser-agent: *Crawl-delay: 10# Allowable IndexAllow: /*?p=Allow: /media/# DirectoriesDisallow: /404/Disallow: /app/Disallow: /cgi-bin/Disallow: /downloader/Disallow: /includes/Disallow: /js/Disallow: /lib/Disallow: /magento/# Disallow: /media/Disallow: /pkginfo/Disallow: /report/Disallow: /skin/Disallow: /stats/Disallow: /var/# Paths (clean URLs)Disallow: /index.php/Disallow: /catalog/product_compare/Disallow: /catalog/category/view/Disallow: /catalog/product/view/Disallow: /catalogsearch/Disallow: /checkout/Disallow: /control/Disallow: /contacts/Disallow: /customer/Disallow: /customize/Disallow: /newsletter/Disallow: /poll/Disallow: /review/Disallow: /sendfriend/Disallow: /tag/Disallow: /wishlist/# FilesDisallow: /cron.phpDisallow: /cron.shDisallow: /error_logDisallow: /install.phpDisallow: /LICENSE.htmlDisallow: /LICENSE.txtDisallow: /LICENSE_AFL.txtDisallow: /STATUS.txt# Paths (no clean URLs)Disallow: /*.js$Disallow: /*.css$Disallow: /*.php$Disallow: /*?p=*&Disallow: /*?SID=# IF YOU DO NOT WISH TO HAVE THE GOOGLE IMAGE BOT SCAN YOUR DOMAIN FOR IMAGES# THEN YOU CAN INCLUDE THE FOLLOWING IN YOUR ROBOTS FILE.# I FOUND THAT MY BANDWIDTH USAGE DROPPED BY A MASSIVE AMOUNT AFTER I GOT RID# OF THE GOOGLE IMAGE BOT. ALL I HAD WAS IMAGE HUNTERS STEALING PRODUCT SHOTS# AND NOT EVEN BROWSING THE SITE.#User-agent: Googlebot-ImageDisallow: /User-agent: Unknown robotDisallow: * User-agent: Unknown robot (identified by empty user agent string) Disallow: *User-agent: Unknown robot (identified by '*bot')Disallow: *User-agent: Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/\-)Disallow: *User-agent: YaBrowserDisallow: *User-agent: CCBot/2.0Disallow: *