Hello all,

As I'm sure many of you noticed we were down for over an hour. The good news is that we're back, the server upgrades that were previously delayed are being completed, and things should be more reliable going forward.

I'd like to apologize for not giving notice that this was going to happen. Unfortunately, the last update from our host indicated that the maintenance wasn't going to happen today. As far as we knew the next attempt at upgrading wasn't due to happen until the 28th. We've brought this up with our host as something that needs to be better communicated going forward. Hopefully the downtime wasn't too much of an annoyance for anyone.

As always, thank you for your patience and understanding.

Hello all,

We're in the process of upgrading to the latest Lemmy release. You can read about what's changed here:

https://crazypeople.online/post/3152222

The upgrade requires taking the site offline for a few minutes, ~~which will happen within 30 minutes of this post~~. The upgrade process is expected to take less than 15 minutes.

Thank you for your patience.

Hello again,

We had about half an hour of unplanned downtime this morning, and there have been a few short times over the last few days when the site was down. I'd like to apologize for that as it is not the experience we intend to deliver.

For reasons that are still being investigated, the PostgreSQL servers at our host were overloaded and required manual restarting rather than the automated restart that should've happened. We're expecting some of this problem to be remedied after this weekends maintenance to upgrade to NVME drives on postgres servers. We're still looking into the problem and talking it over with our host to prevent it from becoming a regular problem now and in the future.

Thanks you for your patience and understanding as we work through these things.

Hello again all,

We're making a few more adjustments to our Pict-rs configuration that require a quick restart. These usually take under a minute from start to finish. During this time the site will be unreachable for local users.

Click here for current UTC Time

Thank you for your patience.

We are making adjustments to our Pict-rs image proxy that will require a quick server restart. These usually take under a minute and aren't expected to cause disruption for more than a few seconds. Sorry for the short notice and thank you for understanding.

Hello again all, hope you're well.

Saturday June 14th crazypeople.online will be going down for maintenance work taking place at our host. During this time the site will be unreachable for local users. The exact shutdown time and expected time to complete the upgrade are still be TBD. This post will be updated when we have more specific information to pass along.

There are several upgrades taking place at this time, most importantly the switch to NVME storage for our PostgreSQL servers. This is expected to drastically increase our I/O throughput, which should provide better performance across all the services we offer as they all interact with PostgreSQL continuously.

Keep an eye on this community for further updates.

~~Hello again all,~~

~~Today we're making some changes to our email configuration that could result in undelivered emails for a short time. To make sure nothing slips through the cracks we've temporarily disabled new user registrations while we setup and test the new config. We expect this process to last a few hours at most, after which registrations will be re-enabled. There will also be a brief restart when the maintenance is completed.~~

~~Thank you for your patience.~~

Updated below.

Hello all, hope you're doing well.

We'll be going down shortly to make some changes to our pict-rs configuration. We've decided to enable the pict-rs image proxy and need to add additional resources in anticipation of the extra load on the servers. There will be a brief restart or two within the next 5-30 minutes.

Thank you for your patience.

Hello again,

We're about to do some upgrades to our Lemmy configuration to keep things running smoothly. During the upgrades the site will be briefly unavailable as services are restarted. Expected downtime is less than 10 minutes. This post will be updated when the upgrades are completed.

Thank you for your patience.

Hello again.

~~Our Pict-rs service is being moved to PostgreSQL as part of our stability upgrades for the site. Within the next few minutes local users will no longer be able to upload images, and the images we host may become unavailable. The expected downtime is less than one hour and only impacts our Pict-rs service.~~

~~This post will be updated when the maintenance is complete.~~

~~Your patience is appreciated.~~

Update:

Maintenance has been completed. If you run into any issues please report them here.

Thank you.

There will be brief downtime to upgrade the hosting package for the site. Postgres will have to be rebooted as well so it will be a few minutes total, probably within the next hour or so.

Sorry for the inconvenience.

Greetings All.

Our host is upgrading some equipment today that will cause us some downtime, some of which is already ongoing. Normally these things will be announced ahead of time but this one slipped through the cracks and we apologize.

Some of the nodes are already back online and we expect the others to come back online within the specified time.

Thanks for understanding!

We will be updating to the latest Lemmy version very shortly. Release notes can be found here:

https://lemmy.ml/post/28296989

Downtime is expected to be approximately 15 minutes.

We experienced a disruption to our Pict-rs service today that caused images we host to temporarily become unavailable and new images unable to be uploaded. Those issues are now resolved and everything should be working normally again.

If you see this or any other problems pop up again don't hesitate to let us know. Thanks.

Hello,

We will be performing some brief site maintenance to add additional resources to the server. Expected downtime is less than 5 minutes as the server restarts. If you're unable to reach the site please try again in a few minutes.

Thank you.

Everything is back up and looks good. Please report anything unusual.

Thanks.

Hello all,

We will be updating to Lemmy 0.19.10 today at 19:00 UTC. That's about an hour from the time of posting.

http://time.is/UTC if you want something more accurate.

Expected downtime is only around 15 minutes. During this time the site will be unreachable, meaning local users won't be able to use the site or use their account via apps. Federated instances shouldn't be impacted and everything should sync back up as soon as the update is completed.

This is a relatively minor update with some new tweaks to help with the ongoing spam problem and a fix for YouTube thumbnail fetching.

Hello again,

Once again we're being bombarded with DMs from spam accounts calling themselves the 'Fediverse Chick' and using the name Nicole. We believe this spam is intended to be some kind of revenge and/or harassment as there is some evidence that Nicole may be a real person who's devices have been hacked. For this reason we've decided to take more drastic measures to block the spammer from reaching our members.

Starting today we will be defederating from instances being used for spam attacks.

We believe the main attack vector for the spammer to be instances with open registrations. By defederating the instances the attacker is using we should slowly be able to wall them out. The vast majority of these instances are extremely small so the impact should be minimal for everyone. If your instance gets blocked and you'd like to continue browsing our content, please ask your instance admins to change their registration policy.

Our suggestion is for all instances to enable captcha for registration as a bare minimum line of defense. Manual approval is even better but adds a lot of extra work for admins of busy instances. If we can increase the time/cost/barrier of entry for the spammer, the attacks should become less frequent.

Lemmy doesn't have a lot of tools for handling these type of attacks and this isn't a step we take lightly. If it turns out defederating is ineffective we'll go back to how things were. Hopefully future versions of Lemmy will feature more robust admin and moderation tools, but for now this is the best we can do to get things stopped quickly. PLEASE use the report feature when you receive new messages from the spammer so that we can continue to block their alts.

Thank you.

Hello again,

Our friends at db0 have come up with another way to block the 'Fediverse Chick' spam. We've implemented the filter detailed here. Please note this won't remove the spam from your inbox. Future messages should display as "Removed" if they contain the filtered phrases.

Please let us know if spam is still slipping through so we can make additional changes as needed. Thanks.

Hello,

We're happy to announce that we've enabled federation with lemmy.world ahead of schedule. After some testing we feel confident that we'll be able to handle the additional traffic even though they are still running an older build of Lemmy.

We've added several of their top communities to our "All" feed found here to get things started. Many more communities should begin showing up over time as our instances begin to interact. Lemmyverse is a great tool for finding communities to subscribe to if you'd like to help things along.

We've done our best to ensure this is a painless experience for our users, but unexpected things can come up. Please let us know of any problems you're having and we'll do our best to help get you sorted.

Thanks.

Hello,

Many of you have probably seen the spam profiles calling themselves the 'Fediverse Chick' by now. They are accounts created on other instances that send annoying DMs asking you to check out their profile. As of today we're implemented a blocking policy on crazypeople.online to try and combat the spam.

Please let us know if any of these messages are still reaching your inboxes after today.

Thank you.

The instance is now online and federating.