I think it's just incompetence (on the part of the scrapers) - been keeping an eye on and having to deal with it for other infra for a while now, and you get a strong impression of very yolo value setting in code-side control: terrible pause/resume flow control logic, fuckall in the way of actual load spread, fuckall sensible backoffs, etc

I'm reasonably certain that a lot of it is written in the lowest-effort ways possible with extremely shit implementation quality, because their focus is purely on getting as much data as quickly as possible and they don't give a single fuck to be good netizens

lemmy falls over at some pitifully low number of hits per second, it's just scraper shitheads

spammers are stupid