68
Creepy crawlies (people.kernel.org)
submitted 1 week ago by to c/opensource@programming.dev
20 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 1 week ago* (2 children)

They're not scraping to cache or store, they're operating as an agent - scraping or single user requests.

Which is obviously bad and damaging, especially on their scale and on repeatedly fetched websites that they could be caching.

Google indexed the entire web. It's baffling that such indexing is not the norm on these huge providers.

Just my interpretation anyway.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 week ago* (last edited 1 week ago) (1 child)

    I considered this, but would agents really ask for single commits with such frequency? They tend to get individual files via HTML, or do a git clone if they needed commit history for some reason.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 week ago

    Simple agents won't understand git. They can make web requests. If they want to check for reference code/source code, I assume they would request the rendered html.

    I assume the frequency comes from many people asking various things, and the agents in the background pulling this data.

    I'm not sure whether the scale-to-load ratio is plausible, because I lack the numbers, but it doesn't seem implausible to me that various agents for various prompts and tasks from various providers for many people repeatedly make these requests to a degree that significantly impacts the hoster.

  • permalink
  • fedilink
  • source
  • parent