41
ELI5: Why do AI web scrapers take down sites? (lemmy.world)
submitted 6 hours ago by to c/explainlikeimfive@lemmy.world
24 comments fedilink hide all child comments
 

I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an ⊛ to webmasters?

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 6 hours ago (3 children)

Because a -r is likely what they are doing. Recursively checking everything every 3 microseconds

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 4 points 5 hours ago (2 children)

    So they're just checking as fast as possible to get every update as fast as possible?

    This still sounds like idiot design, scraping hard enough to take down sites. It's like cutting open the goose that lays lead eggs in the hope that you can get more lead and convince people that lead is better than gold.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 5 hours ago

    While this is idiotic and counter productive long term, the thing to remember, it's not just one scraper, it's hundreds of them, simultaneously trying to outscrape the other, each trying to hoover up as much as possible as fast as possible.

    If that's not enough, the Facebook scrapers hit the same URL over and over again, even when that URL returns an error. Their approach to web scraping is essentially: "Fuck you, deal with it", even if the website is run by a small business that cannot financially justify more bandwidth or server capacity.

    I've seen this process bring sites down, to the point of actively having to block all non-human requests, just to be able to serve customers. This of course doesn't work long term because the bots continue to probe and morph.

  • permalink
  • fedilink
  • source
  • parent