15
Forgejo Scraping Protection: Nginx and Anubis (ayo.tokyo)
submitted 2 days ago* (last edited 2 days ago) by to c/selfhost@lemmy.ml
6 comments fedilink hide all child comments
 

This is a little bit of a wall of text but I hope it will be a nice sample for such a setup.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 days ago (3 children)

I haven't heard anything like that. If so, hope we get a source.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 day ago (2 children)
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 1 point 15 hours ago* (last edited 14 hours ago) (1 child)

    Thanks for the sources, reading them make me realize : MAYBE I have been protected so far because my website is built using InfernoJS and I didn't realize Forgejo doesn't requires Javascript.

    So the spams are more likely to appear from the "low-effort" bots that don't run a Javascript engine. The problem is that the website has only 5 static pages (the JS framework allows better server efficiency and language/theme switch) so spam were never an issue (you can get the whole websites in 10 requests).

    I am thinking about adding a fail2ban rule on the 503 answers triggered by nginx rate limit. This should but this would once again only protected against low-effort bot that don't calculate the rate limit.

    Experience will tell how good of a protection this is.

    Let me add that the website is still not using Anubis, this is only for the public forgejo instance. Also any published software is mirrored on codeberg. I think visitors of the self-host git repository is very niche and specialized, for any one but me looking at this forgejo instance should be more of looking my work or backup of the published work than anything, so very niche.

    PS : loving the runtimewire article concluding "instead of repeatedly charging the humans trying to read the page" when they have this

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 hour ago

    For me, every single request is coming from a different random IP address (different ASN, different country, etc.) using a random plausible User-Agent. There's no way to discern them as bot requests. And since every request is coming from a random source, things like fail2ban or CrowdSec don't really work. You can block a few major ASNs but there will still be lots of requests making it through.

    And these things work by going through a list of URLs previously collected. About a week ago I've configured a redirect for ALL requests to my personal Gitea instance to a zip bomb. It's been 7 days now where the website was inaccessible and I still see requests for deep links, e.g. specific commits or lists of pull requests. These things are relentless. That's also one of the arguments against Anubis - those scrapers just don't care.

    I've found some comments that at least a few ISPs check AbuseIPDB.com and cancel their customer's contracts if their IP appears in that database. So, I've now resorted to occasionally collect those IPs hammering my poor Raspberry Pi and reporting them to the AbuseIPDB. If people are unknowingly running a "residential proxy" (why don't we call them "open proxies" anymore?) somebody needs to let them know, I guess.

  • permalink
  • fedilink
  • source
  • parent