this post was submitted on 05 Jul 2023
26 points (100.0% liked)
Lemmy Server Performance
1 readers
1 users here now
Lemmy Server Performance
lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Using Couchbase as your eventual consistency database is perfect for this scenario. It's designed for this type of thing. Even if systems are offline for a few days they will queue up and replicate when they come back online. Cruise ships use it for this very reason.
Can you provide a link? Only thing I see on Couchbase is for NoSQL databases.
It is a NoSQL database but the SQL syntax is ANSI SQL compliant. If you moved the queues to Couchbase and let it handle the replication and consistency, you wouldn't have to code for it.
Thanks for checking.
We don't need full database replication. We are just replicating activities that other servers are subscribed to. So the Comments table, only some of the rows will be "replicated". Not sure if/how CouchBase handles this.
Buckets are like databases in traditional relational databases. Here's the doc on how to filter which documents get replicated.
https://docs.couchdb.org/en/stable/replication/intro.html#controlling-which-documents-to-replicate
Normally with XDCR you can specify which documents to replicate out of a bucket. It doesn't have to be the entire bucket. So if you had certain types (comments, upvotes, etc) then only those would sync when the target comes online.
I did check into Apache CouchDB, the open source upstream, and replication is there. We use Enterprise Couchbase at work and it's a dream but there are some tools that I use that only use Apache CouchDB (Inkdrop for example). It's worth looking into.