I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an ⊛ to webmasters?

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 2 hours ago* (last edited 2 hours ago)

the conspiracy theorist in me wonders if the AI chucklefucks are using some manner of malware to route their traffic through botnets these days

Not a conspiracy at all: it’s a whole industry called “resnet proxies” that considerably predates LLMs.

There’s a whole good, bad, and ugly to it; they’re invaluable as a privacy and public-access tool, but also great for blackhats and ticket scalpers. Similarly, the proxy nodes come from a full spectrum of sources ranging from normies getting paid kickbacks to plug a box into their router, to pseudo-ISPs serving ghost clients, to malware farms, and everything in between.

  • source
  • parent