39
ELI5: Why do AI web scrapers take down sites? (lemmy.world)
submitted 5 hours ago by to c/explainlikeimfive@lemmy.world
23 comments fedilink hide all child comments
 

I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an ⊛ to webmasters?

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 3 hours ago* (1 child)

Ordinary search crawlers are throttled and they do stuff like respect your robots.txt and sitemap.xml (directives that tell the crawler what links to scan or not and how frequently). These naive crawlers hit everything at max speed, and on a content-heavy site like lemmy that can easily take out a server.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •