74
submitted 1 day ago by [email protected] to c/[email protected]

On June 27:th, the https certificate for lemmy.today was supposed to renew itself. Its set up to automatically renew every 3 months, and its been working perfectly since the site was started over 2 years ago.

But this time it failed to do so, due to a syntax error in the web server config that was not noticed until this incident took place.

Its been fixed now but took more than a day unfortunantly. Sincere apologies for the long delay - it was because I was on vacation in Madeira and in the process of travelling home that day.

Life finds a way to kick us in the balls sometimes.

top 22 comments
sorted by: hot top new old
[-] [email protected] 13 points 1 day ago

Thanks! Happy to know it got resolved, and I hope you had a nice vacation!

[-] [email protected] 11 points 1 day ago

It was very nice and very much needed to get a break from work for a while. Thanks for your understanding!

[-] [email protected] 3 points 1 day ago

May I suggest signing up for hetrixtools.com

Free monitoring service that would have alerted you some weeks before the certificate expired.

[-] [email protected] 1 points 21 hours ago

Its an auto-renewing cert so it "always" works, but in this case there was a syntax error in the web server config, causing it to fail to actually reload the new cert. Will see what we can do to make that not happen again!

[-] [email protected] 2 points 20 hours ago

That cert is valid for 90 days at a time and is renewed every 60 days. So having an alert if the current certificate has less than 28 days remaining will give you an early warning that something went wrong with automation.

[-] [email protected] 2 points 19 hours ago

Ah yeah, thats a good idea. Thanks!

[-] [email protected] 9 points 1 day ago

Yayyy we’re back up! Thanks for the update. And we totally understand that you have a life outside of here lol :p

[-] [email protected] 7 points 1 day ago

Enjoy the vacation, appreciate you so much (:

[-] [email protected] 7 points 1 day ago

Was really worried for a bit that y'all just shut the instance down. I was thinking "at least give me time to migrate my Furryosa pics!!!"

[-] [email protected] 10 points 1 day ago

We will never shut it down just like that... It would be incredibly rude and insensitive towards you guys and we just dont operate that way as people.

This place is supposed to be very different from big tech platforms, and that means we see users as people to start with.

See you guys around!

[-] [email protected] 4 points 1 day ago

Chad energy having mfing mod!

[-] [email protected] 4 points 1 day ago

Thanks so much man.

[-] [email protected] 2 points 1 day ago

Life finds a way to kick us in the balls sometimes.

BOHICA. :p

old.lemmy.today (and all the other alternative interfaces listed in the sidebar) are still having issues. They all seem to be pointing to the default interface instead of their respective alternatives. They're also throwing cert errors, probably because the cert is now for "lemmy.today" instead of "*.lemmy.today"

[-] [email protected] 1 points 21 hours ago

Will be fixed today also, thanks!

[-] [email protected] 1 points 2 hours ago

You da man! Working great now!

[-] [email protected] 4 points 1 day ago

When I got certificate error, I got really surprised. Nowadays no one does it manually. That was the only part made me a little bit concerned. Thanks for having this instance. I really appreciate the effort.

[-] [email protected] 3 points 1 day ago

Thanks for looking at it on your vacation! Sorry it came up then!

[-] [email protected] 2 points 1 day ago

I was worried, so I checked the error. And it made me confident something had gone wrong with autorenewal.

Given that my job had a days-long partial outage due to CLOCK SKEW between nodes, a cert renewal issue is positively understandable.

Glad you were able to get it sorted out, but hope you have some more vacation to get back to!

[-] [email protected] 4 points 1 day ago

Oh man, clock skew between servers totally screwed me about two years ago because the servers were load balanced traffic and applied their server timestamp to realtime data - which in some cases led to entries being out of order. (never mind all the other issues with this implementation)

[-] [email protected] 4 points 1 day ago

Understandable. Have a nice day!

[-] [email protected] 2 points 1 day ago

Let’s encrypt?

[-] [email protected] 1 points 1 day ago

I too thought maybe the instance shut down. I'm glad it didn't. No worries, I hope your vacation went well. I'll survive a few days without social media

this post was submitted on 28 Jun 2025
74 points (98.7% liked)

Announcements

305 readers
59 users here now

Announcements about system updates or other things related to this instance.

founded 2 years ago
MODERATORS