this post was submitted on 12 Jul 2023
63 points (100.0% liked)

Reddthat Announcements

692 readers
1 users here now

Main Announcements related to Reddthat.

founded 1 year ago
MODERATORS
 

Unfortunately we had an issue updating to the latest version of Lemmy. This caused database corruption and we had to roll back to a backup that was unfortunately from 2 days ago!

An abundance of caution in the wrong error resulted in me wanting to push this change out to help with the exploit relating to custom emoji. As other instances were attacked, I didn't want reddthat to fall to the same issue.

I am so sorry !

I have since updated our deployment scripts to automatically take a back up before all updates, as well as investigating why our cron backups didn't work as we expected them too.

We also lost all new accounts that were created over the past 2 days, all new posts, and all subscriptions. I'm very sorry about that and hope that you will reregister with those account details again, repost the memes and pictures that make our server great.

The good news is that we have updated to the latest version of Lemmy and everything is back to "normal".

Fixing Deployments

I'm going to use some of our money for a dedicated development server as our current deployment testing is clearly lacking.
Unfortunately that will have to wait until the weekend when I have some extra time to put into developing these new processes.

Tiff

top 14 comments
sorted by: hot top controversial new old
[–] [email protected] 22 points 1 year ago (1 children)

That sucks, but thanks for the transparency, Tiff. Good call on the auto-backup and dev environment, cool that you have the resources to do that now.

[–] [email protected] 10 points 1 year ago

I've sorted the auto-backup now!

[–] [email protected] 18 points 1 year ago

Growing pains. Part of me adores how janky the Fediverse still is. :D

[–] [email protected] 16 points 1 year ago (1 children)

It’s all good brother cheers from the fediverse

[–] [email protected] 6 points 1 year ago
[–] [email protected] 12 points 1 year ago

Appreciate the effort. Can we get a pinned post next time we have an update incoming and an ETA for how long the server will be down? Thanks.

[–] [email protected] 12 points 1 year ago (1 children)

Two days is nothing compared to the misery that would come from either a compromised server or a botched upgrade.

[–] [email protected] 6 points 1 year ago

Yes. Unfortunately it should never have happened. But at least it was only 2 days, and not weeks!

[–] [email protected] 9 points 1 year ago (1 children)

No need to apologize, things happen and it's a learning curve! 🥰

[–] [email protected] 10 points 1 year ago (1 children)

When you have now... 500! active users and over 2.8k user accounts I feel somewhat responsible for people having a good experience. When have you used a service which lost data twice in a week! I'm annoyed in that these things are easily avoidable but they are causing issues. Moving forward I'll be putting in better more checks around deployments to make sure these cannot happen again.
I keep putting too much trust into the upgrade process when new versions come out expecting it to work without issues.

[–] [email protected] 16 points 1 year ago

I really appreciate the apology, and every web admin and sys admin has either made the same mistake, or watched someone else make it. I could absolutely see my younger self creating an instance of Lemmy and doing the exact same.

The fact that you acknowledged the mistake, learned from it, and communicated the improvements are all great signs that we're in the right place!

Thanks for all you're doing!

[–] [email protected] 7 points 1 year ago

Very sad, was having some great conversation that was wiped. Hope this doesn't happen too often but I understand.

[–] [email protected] 5 points 1 year ago (1 children)

I was wondering why I couldn't login with my creds anymore... 🤣

[–] [email protected] 8 points 1 year ago

I'm so sorry! But I'm super grateful you came back