▲ 45 ▼ Aggressive AI scrapers are making it kinda suck to run wikis (weirdgloop.org) submitted 3 months ago by lemmydividebyzero@reddthat.com to c/technology@lemmy.zip 3 comments fedilink hide all child comments
[–] _deleted_@aussie.zone 8 points 3 months ago (2 children) iocaine doesn’t stop them, but it uses minimal resources and makes me feel better about serving pages to them. permalink fedilink source hideshow 4 child comments replies: [–] undefinedTruth@lemmy.zip 7 points 3 months ago Never heard of it, but I see Anubis pretty widely adopted especially among open source projects. permalink fedilink source parent [–] algernon@lemmy.ml 1 point 3 months ago It can stop them nowadays, by firewalling some of the crawlers off. The reason it doesn't stop them by default is because it serves them poisoned URLs, which it can later identify if the crawlers come back riding a headless Chrome. But once they do that, and hit a poisoned URL, there's little reason to let them wander in an endless maze further: serve one request, and block the IP. I've been running that on my own infra, and my daily number of requests went down from ~50+ million to... 2 million. permalink fedilink source parent
[–] undefinedTruth@lemmy.zip 7 points 3 months ago Never heard of it, but I see Anubis pretty widely adopted especially among open source projects. permalink fedilink source parent
[–] algernon@lemmy.ml 1 point 3 months ago It can stop them nowadays, by firewalling some of the crawlers off. The reason it doesn't stop them by default is because it serves them poisoned URLs, which it can later identify if the crawlers come back riding a headless Chrome. But once they do that, and hit a poisoned URL, there's little reason to let them wander in an endless maze further: serve one request, and block the IP. I've been running that on my own infra, and my daily number of requests went down from ~50+ million to... 2 million. permalink fedilink source parent
all 3 comments