43
ELI5: Why do AI web scrapers take down sites? (lemmy.world)
submitted 7 hours ago by to c/explainlikeimfive@lemmy.world
27 comments fedilink hide all child comments
 

I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an ⊛ to webmasters?

you are viewing a single comment's thread
view the rest of the comments
[–] 11 points 6 hours ago (3 children)

The main issue with AI scrapers is that there isn’t just one, there are dozens or hundreds of them, all doing the same thing. Most websites would handle being hammered by one or two bots just fine, but crash under the load of dozens of bots constantly sending requests.

On top of that, many of them are very badly (vibe)coded and get stuck in endless loops, requesting the same pages over and over again. At my job we IP-blocked an aggressive data-harvesting bot from our service, and the thing got stuck requesting the same page over and over again every 8 seconds, despite receiving the same error code every time. This continued for weeks.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] 11 points 6 hours ago* (last edited 6 hours ago) (2 children)

    A whole eight seconds? Talk about lucky. We have misbehaving bots that attempt to hit the same page or small handful of pages on our site 50-100 times simultaneously, repeating this every couple of seconds. And they have tons of IP addresses they do it from so you don't see the same IP in your logs more than a couple of times a day. They seem to be doing it specifically to attempt to evade simple IP detection in order to make it harder to block them. And there aren't that many IPv4 addresses in the world anymore, so the conspiracy theorist in me wonders if the AI chucklefucks are using some manner of malware to route their traffic through botnets these days. Stealing people's bandwidth and not just their copyrighted content. It'd certainly be on brand.

    For my block lists I have to look at patterns of behavior now instead, and even that's not a sure fire thing. It'd be easy to block a legitimate user who just so happens to be trying to load the same page/product at the same time the clankers are fixated on it.

    Putting the entire thing behind Cloudflare helped a lot. Unfortunately, I just got an email from them which seems to imply that by default they're going to start letting more bots through, which seems like a counterproductive move...

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 6 points 5 hours ago* (last edited 4 hours ago)

    the conspiracy theorist in me wonders if the AI chucklefucks are using some manner of malware to route their traffic through botnets these days

    Not a conspiracy at all: it’s a whole industry called “resnet proxies” that considerably predates LLMs.

    There’s a whole good, bad, and ugly to it; they’re invaluable as a privacy and public-access tool, but also great for blackhats and ticket scalpers. Similarly, the proxy nodes come from a full spectrum of sources ranging from normies getting paid kickbacks to plug a box into their router, to pseudo-ISPs serving ghost clients, to malware farms, and everything in between.

  • permalink
  • fedilink
  • source
  • parent