15
Forgejo Scraping Protection: Nginx and Anubis (ayo.tokyo)
submitted 2 days ago* (last edited 2 days ago) by to c/selfhost@lemmy.ml
5 comments fedilink hide all child comments
 

This is a little bit of a wall of text but I hope it will be a nice sample for such a setup.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 days ago (4 children)

Isn’t Anubis useless against AI scrapers and just annoying people by now?

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] [S] 3 points 2 days ago

    The principle is pretty simple : use a little bit of compute. If you're a bot and want to visit many sites the cost will add up, if you are a human going to a website the 1 second cost for the initial page is not a big deal.

    Also notice the difference with the extremely annoying cloudflare checkbox asking if you are a human that became ubiquitous lately. I'd rather wait a second.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 2 days ago (2 children)

    I haven't heard anything like that. If so, hope we get a source.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 day ago (1 child)
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 1 point 12 hours ago* (last edited 12 hours ago)

    Thanks for the sources, reading them make me realize : MAYBE I have been protected so far because my website is built using InfernoJS and I didn't realize Forgejo doesn't requires Javascript.

    So the spams are more likely to appear from the "low-effort" bots that don't run a Javascript engine. The problem is that the website has only 5 static pages (the JS framework allows better server efficiency and language/theme switch) so spam were never an issue (you can get the whole websites in 10 requests).

    I am thinking about adding a fail2ban rule on the 503 answers triggered by nginx rate limit. This should but this would once again only protected against low-effort bot that don't calculate the rate limit.

    Experience will tell how good of a protection this is.

    Let me add that the website is still not using Anubis, this is only for the public forgejo instance. Also any published software is mirrored on codeberg. I think visitors of the self-host git repository is very niche and specialized, for any one but me looking at this forgejo instance should be more of looking my work or backup of the published work than anything, so very niche.

    PS : loving the runtimewire article concluding "instead of repeatedly charging the humans trying to read the page" when they have this

  • permalink
  • fedilink
  • source
  • parent