285
Aggressive AI scrapers are making it kinda suck to run wikis (weirdgloop.org)
submitted 3 months ago by to c/technology@lemmy.world
22 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 23 points 3 months ago (7 children)

I wonder if it would be possible to filter traffic by limiting the rate at which links can be followed after the initial connection.

I suppose one problem with that is it would also limit the crawlers you want, like search engine indexing. Maybe if enough sites were doing this it would generate some pressure on the AI companies to behave better.

  • permalink
  • fedilink
  • source
  • hideshow 7 child comments
    •
  • [–] [S] 8 points 3 months ago (1 child)

    If they have 10k URLs and in parallel find out all links via different IPs, they get new links.

    When they do that again with the new links, there is no connection between visits, because the IPs differ. It looks like someone else is requesting what is behind the link.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 5 points 3 months ago (4 children)

    The issue with wiki's and source forges is that there is a maze of links to all past versions of everything, each generated on demand from a cpu-expensive database query. You basically have to limit the pages anonymous users can spider into. Forgejo has a setting to block expensive pages from non-logged in users for example.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 4 points 3 months ago (1 child)

    I could see maybe caching that and providing it to a not-clearly-human user if it is in cache. That lets someone do something like link to a particular version of a file in a discussion here on the Threadiverse. The first user loading it will cause it to be cached.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 3 months ago (1 child)

    Probably can configure anubis to require challenge that is proportional to the CPU time needed to render each page?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 3 months ago

    That seems to be a lot of peoples approach, but if they cared about time or bandwidth they wouldn't be spidering Dow into your commit history multiple times a day. They have more patience and resources than your human readers.

  • permalink
  • fedilink
  • source
  • parent