Hello there!
It has been a while since our last update, but it's about time to address the elephant in the room: downtimes. Lemmy.World has been having multiple downtimes a day for quite a while now. And we want to take the time to address some of the concerns and misconceptions that have been spread in chatrooms, memes and various comments in Lemmy communities.
So let's go over some of these misconceptions together.
"Lemmy.World is too big and that is bad for the fediverse".
While one thing is true, we are the biggest Lemmy instance, we are far from the biggest in the Fediverse. If you want actual numbers you can have a look here: https://fedidb.org/network
The entire Lemmy fediverse is still in its infancy and even though we don't like to compare ourselves to Reddit it gives you something comparable. The entire amount of Lemmy users on all instances combined is currently 444,876 which is still nothing compared to a medium sized subreddit. There are some points that can be made that it is better to spread the load of users and communities across other instances, but let us make it clear that this is not a technical problem.
And even in a decentralised system, there will always be bigger and smaller blocks within; such would be the nature of any platform looking to be shaped by its members.
"Lemmy.World should close down registrations"
Lemmy.World is being linked in a number of Reddit subreddits and in Lemmy apps. Imagine if new users land here and they have no way to sign up. We have to assume that most new users have no information on how the Fediverse works and making them read a full page of what's what would scare a lot of those people off. They probably wouldn't even take the time to read why registrations would be closed, move on and not join the Fediverse at all. What we want to do, however, is inform the users before they sign up, without closing registrations. The option is already built into Lemmy but only available on Lemmy.ml - so a ticket was created with the development team to make these available to other instance Admins. Here is the post on Lemmy Github.
Which brings us to the third point:
"Lemmy.World can not handle the load, that's why the server is down all the time"
This is simply not true. There are no financial issues to upgrade the hardware, should that be required; but that is not the solution to this problem.
The problem is that for a couple of hours every day we are under a DDOS attack. It's a never-ending game of whack-a-mole where we close one attack vector and they'll start using another one. Without going too much into detail and expose too much, there are some very 'expensive' sql queries in Lemmy - actions or features that take up seconds instead of milliseconds to execute. And by by executing them by the thousand a minute you can overload the database server.
So who is attacking us? One thing that is clear is that those responsible of these attacks know the ins and outs of Lemmy. They know which database requests are the most taxing and they are always quick to find another as soon as we close one off. That's one of the only things we know for sure about our attackers. Being the biggest instance and having defederated with a couple of instances has made us a target.
"Why do they need another sysop who works for free"
Everyone involved with LW works as a volunteer. The money that is donated goes to operational costs only - so hardware and infrastructure. And while we understand that working as a volunteer is not for everyone, nobody is forcing anyone to do anything. As a volunteer you decide how much of your free time you are willing to spend on this project, a service that is also being provided for free.
We will leave this thread pinned locally for a while and we will try to reply to genuine questions or concerns as soon as we can.
Thanks for all your amazing work! I know just enough about SQL to know I know next nothing, but could someone intelligent explain how databases are publicly accessible for anyone to be able to make queries?
They don't need to be. When you're posting a comment, that's a database query. Not from you directly, but you're submitting a comment, which tells the frontend to tell the backend to tell the database to save that comment.
Now do that a thousand times and you created a thousand database queries. Now do something more elaborate, like filtering search results or something, and you put a bit more load on the database.
And apparently there seem to be some queries that a user can create that cause issues if submitted by the thousands.
Oh wow, so someone could literally just post a comment somewhere that says select all or drop tables and it’ll run?
Probably a over simplification, but I guess someone could easily make a script that does that thousands of times.
No, he just means that If you are creating a comment, you are creating entry with a comment in the database indirectly. What you are mentioning is an sql injection attack, but majority of the systems are built with enough brain to prevent it. https://en.m.wikipedia.org/wiki/SQL_injection
Normally there's a separation between raw SQL and commands being issued by the UI to avoid what you're describing (SQL Injection).
For example, the UI might call
/posts/newwith a JSON body describing the new post, this might look like,The Lemmy server then going to receive that "message" from the UI and use it to execute something like
INSERT INTO posts SET title='my post' body='thoughts and stuff, I dunno' author_id='my user id'This is fine.
What's actually the problem is if you do something more complex, like if the UI queries for all posts by a set of authors where the time they were submitted was between some specific date range with contents matching blah blah blah and some other criteria. Make a the right query and you might hit a situation where the database behind the Lemmy server doesn't have the data organized (think, like, a library; how things are organized by author name, or book title) in a way that makes finding matching results an easy thing to do; and instead the database need to look over every. single. post. multiple times to gather up the right results.
Now do this 1000s of times in quick succession.
Theoretically yes, that would be an sql injection. SQL Injection is preventable through good practice, but with added complexity of the application comes more chances for things to be missed.
In some ways, it's good that lemmy is going through this now and getting vulnerabilities exposed sooner than later.
They're not, what the attackers are most likely doing is just repeating some regular operation that they know is very costly to the database. It could be something as simple as just spamming the upvote button, for example. Write a script to do that frequently enough and boom, instance taken down.
It's probably not, but let's say that replying to a comment deep in a thread required an update to every parent comment (very unlikely), if the attackers knows this they could trigger that very expensive SQL query many times very quickly by letting some bots comment many times to create a very deep comment chain and force the server to make the queries. If the server didn't, it would either mean that no one could comment in that way it would have to know what is and isn't an attack which isn't easy.
I doubt they're hitting the DB directly. More likely calling a public API endpoint which makes the query.