Crawlers have never been a problem for me as my internet subscription is unlimited. My experience comes from being the one crawling sites and bypassing the CF challenge