285
Aggressive AI scrapers are making it kinda suck to run wikis (weirdgloop.org)
submitted 3 months ago by to c/technology@lemmy.world
22 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 3 months ago (4 children)

The issue with wiki's and source forges is that there is a maze of links to all past versions of everything, each generated on demand from a cpu-expensive database query. You basically have to limit the pages anonymous users can spider into. Forgejo has a setting to block expensive pages from non-logged in users for example.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 4 points 3 months ago (1 child)

    I could see maybe caching that and providing it to a not-clearly-human user if it is in cache. That lets someone do something like link to a particular version of a file in a discussion here on the Threadiverse. The first user loading it will cause it to be cached.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 3 months ago (1 child)

    Probably can configure anubis to require challenge that is proportional to the CPU time needed to render each page?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 3 months ago

    That seems to be a lot of peoples approach, but if they cared about time or bandwidth they wouldn't be spidering Dow into your commit history multiple times a day. They have more patience and resources than your human readers.

  • permalink
  • fedilink
  • source
  • parent