this post was submitted on 23 Jul 2023

240 points (99.2% liked)

Lemmy Server Performance

1 readers

1 users here now

lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.

founded 1 year ago

MODERATORS

[email protected]

240

GREAT NEWS about Lemmy Server Performance, another major SQL mistake has been discovered today: every single comment & post create (INSERT) is updating ~1700 rows in the site_aggregates table (lemmy.ml)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

32 comments fedilink hide all child comments

Details here: https://github.com/LemmyNet/lemmy/issues/3165

This will VASTLY decrease the server load of I/O for PostgreSQL, as this mistaken code is doing writes of ~1700 rows (each known Lemmy instance in the database) on every single comment & post creation. This creates record-locking issues given it is writes, which are harsh on the system. Once this is fixed, some site operators will be able to downgrade their hardware! ;)

top 27 comments

sorted by: hot top controversial new old

[–] [email protected] 38 points 1 year ago (1 children)

This is fascinating

My biggest takeaway from reading through the GitHub comments though is that it seems like no one actually knows where much of the SQL comes from? As in it's possible that the bug in question is just one manifestation of old, handwritten Postres code that may or may not be optimized (Or even logical?).

I don't mean this in a critical way, as things like this are bound to happen in an open-source, federated world. However, I would think a comprehensive audit of the Lemmy Postgres triggers, queries, etc could potentially save us all from some future headaches.

[+] [email protected] -46 points 1 year ago (4 children)

That's not fascinating, that's depressing. Lemmy team lacks development skill.

[–] [email protected] 33 points 1 year ago* (last edited 1 year ago) (1 children)

I am always fascinated with these types of comments, specifically for a free and open-source software. There are lemmy instances supporting hundreds of thousands of users and trafic, feedback from both server owners and lemmy devs is almost instantaneous.

A platform like lemmy requires client side knowledge to build both desktop and mobile UI (that are performant), it requires ActivityPub knowledge to integrate with the Fediverse, it requires backend knowledge to build APIs for 100% feature compatibility with 3rd party apps. It requires DB knowledge to optimize queries, it requires devops / platform knowledge to deploy it.

And all of this is built in public.

BuT LEMmY tEaM lAcks dEvEloPmenT SkiLL – sure buddy.

[–] [email protected] 13 points 1 year ago

Or there's just room for improvement and optimization, as each developer has its strengths and weaknesses, as any other professional, and a system like lemmy is very complex and really requires to cover a lot from backend to front end.

And there used to be only 2 developers.

I once check a open source implementation of a niche product from Microsoft, and it was a nightmare of unoptimized code. And Microsoft spent a lot of development resources there.

Creating lemmy as 2 people job is quite impressive. Luckily now there are resources for optimization

[–] [email protected] 11 points 1 year ago

This is such a dumb take. Using a database efficiently is not some binary, once-off thing: you build what works based on the data you have at the time. When it works, you move on to other features. It takes analysis of real operation over time to find the bottlenecks, and discipline to focus on fixing the things that will have the most benefit to your users.

There are many successful tech companies who introduce features that create dogshit performance impacts regularly. They work because there are people looking at metrics and catching issues to fix. This is healthy.

[–] [email protected] 7 points 1 year ago (1 children)

Luckily with open source new team members or forking can address that.

[–] [email protected] -2 points 1 year ago

Yep.

[–] [email protected] 36 points 1 year ago (1 children)

Holy hell. Post this to one of the programming-related communities. That is interesting.

[–] [email protected] 1 points 1 year ago (1 children)

Not that interesting when it turns out it was an ORM artifact.

[–] [email protected] 1 points 1 year ago

True. I have experienced quite a few of those... But, still interesting nonetheless.

[–] [email protected] 29 points 1 year ago* (last edited 1 year ago) (1 children)

It's not on every comment, it's mostly triggered on deletions and edits. The problem is actually infintesimally worse and 1700 rows are updated if you delete 3 comments. If you delete more it's exponential and just straight up fails and locks your database.

I'll probably put a patch in there later tonight and then see about a PR unless someone else does.

[–] [email protected] 28 points 1 year ago* (last edited 1 year ago) (1 children)

It’s not on every comment,

My testing with latest code is that it is indeed on every single comment INSERT, every new comment. I have the ability to view my live data while using Lemmy: https://lemmyadmin.bulletintree.com/query/raw_site_aggregates?output=table

Every one of the 1486 rows on that table gets +1 on comment when I post a new comment on my instance.

it’s mostly triggered on deletions and edits

That is not correct. Edits do not change the count of comments column on site_aggregates - because the number isn't changing. Deletes (of a comment or post) in Lemmy are also not SQL DELETE statements, they are just a delete data column in the table. That DELETE PostgreSQL trigger only gets run when a end-user cancels their Lemmy account in their profile.

[–] [email protected] 11 points 1 year ago

Ah Gotcha. That's true, but the cascading issue that causes thousands of inserts happens on a delete.

That table update you're looking at is blazing fast due to the relatively low number of sites, until you run it thousands of times on a delete.

[–] [email protected] 24 points 1 year ago (2 children)

Holy shit

[–] [email protected] 7 points 1 year ago (1 children)

Indeed

[–] [email protected] 8 points 1 year ago

[–] [email protected] 7 points 1 year ago

Mamma Mia

[–] [email protected] 21 points 1 year ago

Man that is some bug, no wonder lemmy had such a rough start performance wise during the reddit migration!

[–] [email protected] 15 points 1 year ago (1 children)

Get some DBA's on the job and Lemmy will be blazing fast.

[–] [email protected] 14 points 1 year ago (2 children)

We have had DBA's, the problem is the Rust code uses ORM and an auto JSON framework that makes tracing the code time-consuming to learn.

[–] [email protected] 3 points 1 year ago

Okay so you may need to refactor here and there to get more performance.

[–] [email protected] 3 points 1 year ago (1 children)

Honestly, ORMs are a waste of time. Why not use sqlx and just hand write the SQL to avoid issues like this.

[–] [email protected] 1 points 1 year ago

In this one case, it was hand-written SQL inside a PostgreSQL FUNCTION that the ORM knows nothing about. But there is a approach in the entire application to have live-data from PostgreSQL for every little thing.

[–] [email protected] 10 points 1 year ago

Wonderful. Time to start running my own instance.

[–] [email protected] 8 points 1 year ago

Big victories :D

load more comments