My arch system is so stable it's boring. Year after year, just runs. Doesn't slow down, doesn't install crapware.
So I dont know. I'm happy with it.
My arch system is so stable it's boring. Year after year, just runs. Doesn't slow down, doesn't install crapware.
So I dont know. I'm happy with it.
Also, while improving server performance and failure modes under load would help, the “badly-written, aggressive scraper bots that ignore robots.txt and are given loads of network resources clobbering servers” is something that affects many, many different Web servers out there. This isn’t a Lemmy problem, nor even just social media problem. It’s a Web problem.
100%. Everyone gets this problem and then they put their site behind cloudflare. But what happens when the entire Internet is behind an american company who can decide which sites it likes and which it doesnt?
Same thing for the Threadiverse. The Threadiverse will let you set up a Threadiverse instance and subscribe to everything, efficiently feed all the posts and comments you want to your instance, the moment they come in. In nice, machine-readable form, rather than in something intended for humans that you have to scrape and post-process. But…it takes more dev effort to set up something specific to the Threadiverse than to just treat it like another website.
Yeah, the web was intended to be this universal format that would always work on any platform and any operating system. But the downside is clear today. Some estimates say that we already have much more bot activity than human activity on the internet. And we have AI creating content for the last few years...
I totally agree with you actually. Maybe there will be some better solution going forward.
@tal@lemmy.today @PapaSkwat@lemmy.today This should be fixed now. It was caused by incorrect caching of the posts. :)
Absolutely. :)
Hi,
I could actually reproduce this myself now:
https://lemmy.today/post/57605044
This link only works sometimes for me too. I have no idea why so will dig into the server logs. :)
I could but there are so many requests, like 30-50 per second, and even if I could cache the login page, I would still spend the bandwidth to send it back to each of those bots. So because of that, and because its not even the main lemmy.today site, I figured its OK to just return 403.
Server load has decreased to almost nothing on old.lemmy.today. During the years we have been online, more and more bots are scraping and now its just ridicolous. :)
It's fine to not give an email address. It's just for the email to let you know that you were approved or denied. So if you put something random there and check if you can login later, it's all good.
I think it's even optional to give it... You can try registering without. :)
Make a PR. :)
Yeah the API should be exempt. Will look into this a bit more in the coming days but wanted to see how well Anubis works on old.lemmy.today first for a while. We probably end up using it on lemmy.today as well. Just didnt want to make a big bang change everywhere on day 1. :)
It sits between your web browser and old.lemmy.today and detects if you are a bot. If you are, your request doesn't pass through Anubis. It's Gandalf saying "you shall not pass".
So it will hopefully cut down on bots scraping the instance. Or it will piss them off even more. :)
They pass dystopian chat control with one hand and does this with the other.