42
ELI5: Why do AI web scrapers take down sites? (lemmy.world)
submitted 7 hours ago by to c/explainlikeimfive@lemmy.world
26 comments fedilink hide all child comments
 

I'd link to some blog posts about this an example, but the site they're from went down a while ago.

At risk of giving anyone ideas, why do LLM training scrapers request pages from sites millions of times per day instead of just doing the equivalent of wget -r https://example.com/ ? If the point is just stealing things people have written, what do they accomplish by wasting a web host's resources beyond being an ⊛ to webmasters?

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 2 points 6 hours ago* (last edited 6 hours ago) (6 children)

But isn't this how ordinary search crawlers and tools like wget also work? I've never heard of them causing these same problems.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 2 points 5 hours ago* (3 children)

    Ordinary search crawlers are throttled and they do stuff like respect your robots.txt and sitemap.xml (directives that tell the crawler what links to scan or not and how frequently). These naive crawlers hit everything at max speed, and on a content-heavy site like lemmy that can easily take out a server.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 3 hours ago (2 children)

    Why do the AI companies' crawlers work differently from traditional ones?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 hour ago* (1 child)

    It’s not necessarily the big companies themselves doing it? But, there’s a push right now to scrape as much data as possible to feed the training of the AI models, and if your business model is entirely based on selling that data you don’t really care who you’re going to piss off. So, in turn these new scraper bots basically behave in the same way that attack bots do where they’ll be behind VPNs and switch IPs and stuff if you block them.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 15 seconds ago

    But WTF is the point of scraping the same data over and over multiple times a second before it even has a chance to change? That's just a waste of resources even on the scrapers' part, because that bandwidth could be used grabbing some other new page instead!

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 6 hours ago (1 child)

    Same difference as being hit by a golf cart rolling down a sleight slope and 100 Semi's going 100mph

    The problem isn't what they're doing, it's the speed and depth. There's no concern for efficiency because they're not paying for hardware and utilities.

    Everything AI is focused on doing asuch as possible as fast as possible, with the hope optimization will happen organically to the point it becomes profitable.

    But it won't.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 5 hours ago

    Correct. Google indexing your web page is rate limited for this reason, and you can even include a directive in your robots.txt to specify your own rate limit if you'd like the intervals to be longer (or shorter). The AI scrapers completely ignore your robots.txt. Except, I am certain, for abusing it as if it were a site map. Anything you list there is simply a target you've revealed to them. ("Hey, robots.txt says we shouldn't crawl /foo/bar.html. That means there's a page there! Let's hammer it with 900 page load requests per second!")

  • permalink
  • fedilink
  • source
  • parent