this post was submitted on 01 Jul 2023
3737 points (97.2% liked)

Lemmy.World Announcements

29098 readers
11 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @[email protected] and @[email protected] for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @[email protected] had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilΓ . That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

(page 5) 50 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 1 year ago

Excellent news! I can browse Lemmy on my phone again, many thanks to you @Ruud and everyone else who worked on this!

[–] [email protected] 4 points 1 year ago

This is really cool that this gets shared. Thanks for all of the info!

[–] [email protected] 4 points 1 year ago

Working well here and can use Jerboa again. Although wefwef is really growing on me!

Edit: couldn't post from Jerboa, got network error. But wefwef worked.

[–] [email protected] 4 points 1 year ago

Is there a way to follow the update status while this instance is down?

I knew the update was coming, so I expected the downtime, but I could not find any info about when it was expected to be up and running again?

Thanks for the great and ongoing work put into managing this endeavor!

[–] [email protected] 4 points 1 year ago

Great stuff!

[–] [email protected] 4 points 1 year ago

Not sure what fixed it (I'm sure though on my side I did nothing since the last failed login) but the login form finally works again!

[–] [email protected] 4 points 1 year ago

Hey man I just want to say thank you for your hard work.

[–] [email protected] 4 points 1 year ago

Even using Tor, the site load-times seem a lot snappier. Exciting times.

[–] [email protected] 3 points 1 year ago

Nice work :)

[–] [email protected] 3 points 1 year ago
[–] [email protected] 3 points 1 year ago

Thanks for all the time and work you put towards making this community better! It's really appreciated!

[–] [email protected] 3 points 1 year ago

Absolutely loving the new UI changes. Thanks for being so transparent with us as well!

[–] [email protected] 3 points 1 year ago

Thanks! Now I can use Connectβ„’ for Lemmy without problems. πŸ‘πŸ™‚

[–] [email protected] 3 points 1 year ago

This is awesome. Was a fun read too. Super cool to see what was going on behind the scenes.

[–] [email protected] 3 points 1 year ago

Let us know where donations can go, suspect a stacked docker-compose will reach limits very quickly

[–] [email protected] 3 points 1 year ago

Hurray! Works well for me so far.

[–] [email protected] 3 points 1 year ago

It seems there is still some performance issues.

Maybe consider a webcache like varnish to take some of the load off?

[–] [email protected] 3 points 1 year ago (1 children)

we created extra lemmy containers to spread the load. (And extra lemmy-ui containers).

Is Rust HTTP server running into thread limits? database connection pooling? All kinds of internal questions bout that solution.

[–] [email protected] 3 points 1 year ago

I don't know rust. But there were 150 database connections setup by lemmy, but only about 15 of them were used, the rest idle.

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago)

Running so many Lemmy instances against the same database doesn't cause race conditions? I wonder why that "just worked" so easily, usually load balancing DB-backed apps is a whole beast on its own.

[–] [email protected] 3 points 1 year ago

Thanks a lot, you’re great

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago) (1 children)

Half the time when I comment, it just spins. :( Edit: Apparently when I comment it posts, but just shows spinning until I manually refresh. Must be on my end.

load more comments (1 replies)
[–] [email protected] 3 points 1 year ago

Thanks for getting the update out!

Definitely weird that it was 1500 instead of some power of 2*100!

Also, when we finally get a 128 thread machine just think of the meme potential!

[–] [email protected] 3 points 1 year ago

I really enjoy how lemmy is growing!

[–] [email protected] 3 points 1 year ago

I really enjoy how lemmy is growing!

[–] [email protected] 3 points 1 year ago

Logging in works now! Also got 2FA enabled without issues.

[–] [email protected] 3 points 1 year ago

Thank you for the hard work! Enjoying Lemmy.world quite a bit. :)

[–] [email protected] 3 points 1 year ago

Looking good πŸ‘

[–] [email protected] 3 points 1 year ago

I survived!

[–] [email protected] 3 points 1 year ago

πŸ™Œ Great work team!!!

[–] [email protected] 3 points 1 year ago

Signed into Jerboa! Thanks devs!

[–] [email protected] 3 points 1 year ago

Just want to say thank you. Your hard work is very much appreciated.

load more comments
view more: β€Ή prev next β€Ί