this post was submitted on 05 Jul 2023
40 points (100.0% liked)

Beehaw Support

2796 readers
1 users here now

Support and meta community for Beehaw. Ask your questions about the community, technical issues, and other such things here.

A brief FAQ for lurkers and new users can be found here.

Our September 2024 financial update is here.

For a refresher on our philosophy, see also What is Beehaw?, The spirit of the rules, and Beehaw is a Community


This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.


if you can see this, it's up  

founded 2 years ago
MODERATORS
 

We have to move to object storage to have efficient image storage. We are currently at 70% of our disk storage.

We are sorry for the short notice and we don't know how long it will take though we believe it will take multiple hours.


Small post mortem :

  1. The migration required us to be down - we thought we could run without images
  2. It took much longer than initially anticipated
  3. It broke at close to 2:30AM - 3 hours into the migration and I couldn't fix it so the server has been down doing nothing for hours now
  4. We're booting back up from the backup made before the migration was attempted, we'll try another strategy.

There might've been some data loss in images, we're looking into it. In the meanwhile, if your profile picture or banner is broken, feel free to re-upload them.


Update :

Hi Beeple!

We're trying this again.

This time, Beehaw should remain available though no pictures will be able to be uploaded. The error will likely be weird because Lemmy will think it is possible but we will block the upload from happening.

We'll take a snapshot of the pictures before the migration in 60 minutes at 21:00 UTC - it should take around one hour, do the migration and testing on our own before shipping on Beehaw.

Once we've resolved all these kinks, Beehaw will momentarily go down and then back up with the migration complete without losing any old pictures.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 10 points 1 year ago (4 children)

Well that was fun.

Didn't go as planned of course, restored from backups, pre migration attempt. Thank you for your patience while we try to get all these moving parts working well together. Sorry for the troubles.

[–] [email protected] 9 points 1 year ago (2 children)

I once caused an AWS outage that impacted 20% of their customers in their largest region. They called my manager to ask why we were performing around 10k writes per second to a bucket. It was fun times

[–] [email protected] 2 points 1 year ago (1 children)

They don't limit that?! I've worked with a lot of AWS services and most have built in rate limits. That's wild lol

[–] [email protected] 7 points 1 year ago (1 children)
[–] [email protected] 4 points 1 year ago (1 children)

lol, that's how rules get made

[–] [email protected] 1 points 1 year ago

I can get into it in more detail if anyone's interested. But basically, they had a rate limit on direct writes, but not a rate limit on cross bucket replication if you connected many buckets to replicate into a single bucket

[–] [email protected] 2 points 1 year ago

That was you?!

(jk, I'm on a different cloud 😂)

[–] [email protected] 4 points 1 year ago

Loads of love. There's always ASCII art.

[–] [email protected] 1 points 1 year ago

I appreciate the late night efforts and the clear communication. For me, Beehaw is a positive place I can visit, but there are other things I can do also, and I have no need for many 9s of uptime here. (I'm trying to reduce any pressure you & others might feel - perhaps not communicated it well tho, hence this addition.)

[–] [email protected] 1 points 1 year ago

Thanks for the update and hope you have less trouble in the future! Don't worry about the downtime I really appreciate that here it's serving a clear purpose unlike Twitter lol