Everybody in this thread is talking about what they're doing, but not a single reply has been able to explain why the bad behavior somehow benefits the companies doing it.

I don't believe the only reason is incompetence; there's got to be somehing else to it.

You are sitting behind a receptionist’s desk with three coworkers. You need to check people into your office for visits.

Because there are four of you, and people are generally good about queuing, your normal operations run smoothly.

Now fifty people walk into the building at once. They don’t really care about checking in, or even visiting the office. They want to know if you have free coffee, a public bathroom, your elevator inspection on file, the date of your last fire inspection, and dozens more inane questions that aren’t what you’re used to handling. They refuse to wait in a line, they’re talking over one another, and if you take more than ten seconds to answer, they walk out of the building but come back to bother you a few minutes later.

Does your office run well, and can you check in a legitimate visitor in a timely manner still?

The latest scrape I saw that took down one of my sites just threw shit at the wall until something stuck. It took one second and did 50+ requests mostly for pages that didn’t exist.

Non-expert here, so conjecture warning.

Loading the info from the page once is fine for training one LLM at one point in time. But there are a bunch of different companies and people doing that, and they probably keep doing it again because they want newer information.

And training is only one aspect of AI scrapers. You also have agents constantly doing websearches and summarizing, synthesizing the info at user request, all day every day.

The main issue with AI scrapers is that there isn’t just one, there are dozens or hundreds of them, all doing the same thing. Most websites would handle being hammered by one or two bots just fine, but crash under the load of dozens of bots constantly sending requests.

On top of that, many of them are very badly (vibe)coded and get stuck in endless loops, requesting the same pages over and over again. At my job we IP-blocked an aggressive data-harvesting bot from our service, and the thing got stuck requesting the same page over and over again every 8 seconds, despite receiving the same error code every time. This continued for weeks.

Short answer: they aren't just grabbing one page. They are hitting every page and link that is available in rapid succession.

As soon as a page loads, the AI goes to a new page. Not just one new page, every link on the page.

It's functionally a ddos attack. Because it quickly spirals exponentially. The limit isn't how fast the AI can scrape, it's how much bandwidth the website server has.

Because a -r is likely what they are doing. Recursively checking everything every 3 microseconds