Yeah hosting just about anything is terrible these days. These AI scrapers just can't act normally, there was nothing wrong with the way GoogleBot and Bing Bot work. They scrape the website, respect robots.txt and nofollow, they rate limit themselves as to not overload the servers. It was just fine.

These days with those AI scrapers they go absolutely ape shit, they issue dozens of requests every second, try to scrape anything and everything. Going so far as to make up urls, just to see if they get lucky. My blocklist is huge and I need to keep updating it all the time. And every now and again one slips through and absolutely slams the server. This causes an alert and I need to act right away. It's fucking terrible.

AI is already shit, why do those companies go out of their way to be even more shit?