16
Forgejo Scraping Protection: Nginx and Anubis (ayo.tokyo)
submitted 2 days ago* (last edited 2 days ago) by to c/selfhost@lemmy.ml
7 comments fedilink hide all child comments
 

This is a little bit of a wall of text but I hope it will be a nice sample for such a setup.

all 7 comments

sorted by: hot top controversial new old
[–] 1 point 5 hours ago

I feel like a simple rate limit set to what your server can handle without issues is a better solution. Or combine it with Anubis at least.

  • permalink
  • fedilink
  • source
    •
    [–] 2 points 2 days ago (5 children)

    Isn’t Anubis useless against AI scrapers and just annoying people by now?

  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] [S] 3 points 2 days ago

    The principle is pretty simple : use a little bit of compute. If you're a bot and want to visit many sites the cost will add up, if you are a human going to a website the 1 second cost for the initial page is not a big deal.

    Also notice the difference with the extremely annoying cloudflare checkbox asking if you are a human that became ubiquitous lately. I'd rather wait a second.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 2 days ago (3 children)

    I haven't heard anything like that. If so, hope we get a source.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 day ago (2 children)
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 1 point 20 hours ago* (last edited 20 hours ago) (1 child)

    Thanks for the sources, reading them make me realize : MAYBE I have been protected so far because my website is built using InfernoJS and I didn't realize Forgejo doesn't requires Javascript.

    So the spams are more likely to appear from the "low-effort" bots that don't run a Javascript engine. The problem is that the website has only 5 static pages (the JS framework allows better server efficiency and language/theme switch) so spam were never an issue (you can get the whole websites in 10 requests).

    I am thinking about adding a fail2ban rule on the 503 answers triggered by nginx rate limit. This should but this would once again only protected against low-effort bot that don't calculate the rate limit.

    Experience will tell how good of a protection this is.

    Let me add that the website is still not using Anubis, this is only for the public forgejo instance. Also any published software is mirrored on codeberg. I think visitors of the self-host git repository is very niche and specialized, for any one but me looking at this forgejo instance should be more of looking my work or backup of the published work than anything, so very niche.

    PS : loving the runtimewire article concluding "instead of repeatedly charging the humans trying to read the page" when they have this

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 7 hours ago

    For me, every single request is coming from a different random IP address (different ASN, different country, etc.) using a random plausible User-Agent. There's no way to discern them as bot requests. And since every request is coming from a random source, things like fail2ban or CrowdSec don't really work. You can block a few major ASNs but there will still be lots of requests making it through.

    And these things work by going through a list of URLs previously collected. About a week ago I've configured a redirect for ALL requests to my personal Gitea instance to a zip bomb. It's been 7 days now where the website was inaccessible and I still see requests for deep links, e.g. specific commits or lists of pull requests. These things are relentless. That's also one of the arguments against Anubis - those scrapers just don't care.

    I've found some comments that at least a few ISPs check AbuseIPDB.com and cancel their customer's contracts if their IP appears in that database. So, I've now resorted to occasionally collect those IPs hammering my poor Raspberry Pi and reporting them to the AbuseIPDB. If people are unknowingly running a "residential proxy" (why don't we call them "open proxies" anymore?) somebody needs to let them know, I guess.

  • permalink
  • fedilink
  • source
  • parent