MangoPenguin comments on What just happened to TheNumbers.com should worry us all

[–] MangoPenguin@piefed.social 13 points 2 months ago (3 children)

How come none of these sites being hit by AI to the point they can't function implement rate limiting?

Rate limiting grouped by IP or UA tends to work pretty well, and normal users have very slow request rates so generally never get impacted.

permalink

fedilink

source

hideshow 3 child comments

[–] JustTesting@lemmy.hogru.ch 5 points 2 months ago (2 children)

Residential proxies are a thing.

Also speaking from experience, especially asia based crawlers will send 5 requests at once from one IP, then 5 from the next etc. and the first IP wont appear again for several hours, making ip based rate limiting useless.

In my case i have honeypot links that block them, i block several data center adress ranges, have an automated whitelist for registered users, use some public blocklists and user agent filtering and that takes care of 99% of it. But it is non-trivial compared to rate limiting.

permalink

fedilink

source

parent

hideshow 2 child comments

[–] MangoPenguin@piefed.social 2 points 2 months ago (1 child)

True, I also run crowdsec which blocks excessive 404s and other unusual requests too, between that and rate limiting I don't seem to have any server overload issues for the most part.

permalink

fedilink

source

parent

hideshow 1 child comment

[–] kmacmartin@lemmy.ca 1 point 2 months ago

In the article they talk about getting hit by millions of different IPs. It's probably on a different level compared to what a random, non-targeted site has to deal with.

A few years back when graphics cards were getting botted for resale, a friend of a friend was buying with bots using a (pricey) rotating list of hundreds of proxy addresses that would each make attempts at a very reasonable looking frequency. I'm picturing something like that, except with much more funding.

permalink

fedilink

source

parent