I'm okay with a few crawlers, but not what's effectively a DDoS attack by AI companies who abuse my resources generating terabytes of traffic and crashing my server while costing me money. I use Anubis now, which sucks from an accessibility standpoint but I'm not dealing with their malicious traffic anymore.