Hey, just wanted to clear up that it's only users from the UK who can't access lemmy.zip directly. They can access the lemmy.zip communities via federation just fine from other instances :)
That's very kind of you ❤️
There is no need for us to defederate, the law is around accessing a specific website and the onligations to ensure the content on that website doesnt fall under the OSA.
It doesn't matter where that content comes from, just that UK users don't see specific content. Therefore we don't show UK users any content.
That's kind of a simplified summary, but hopefully makes sense.
It only impacts lemmy.zip because I am a UK resident and run the site, so am liable under the act. Otherwise I'd do what everyone else has done and ignore it.
Hey, welcome along! :)
I will certainly consider it! :D pre-warning, I am not promising many of them are good 😂
Hey! They are privately owned yes (by me) and currently not in a foundation - donation amounts have never really warranted the cost or complexity of setting anything up.
However, all admins have a level of access to the backend of the site, including SSH, Cloudflare, Hetzner etc. so that the site can be maintained even if I were to vanish.
If I step down, there is sufficient technical knowledge (and equally as importantly, admins with the right ethos!) to keep the site running and to take on the ownership.
I do have a plan to move things into some form of non-profit/community structure when it becomes warranted. That does mean doxxing myself though, so I'm holding off until necessary :)
Appreciate that, thank you 🙏
We had a slightly over-beefy server for our old needs, but now - oof
Welcome aboard! The admin team have been flat out approving the requests (and taking time to enjoy many of the application messages!) so I'm really happy people are noticing this.
While the circumstances are rubbish, we're glad to have you here :)
Yeah donations are an important point - the server is coping with the current levels of traffic, but its certainly not thriving! If we are lucky enough to be chosen by a majority of .ee, we will definitely need to consider an upgrade or additional hardware.
Hey, welcome :)
I think in the region of 150 to 200. For us, thats quite a few :)
Hello all!
Firstly, apologies I missed last month's update. As I will go in to during this month's update, the train has not been smooth sailing, the ship missed the station - for a little while, trying to uncover what has gone wrong has been like unraveling a can of worms that has gone off the tracks.
Because of the length of this update, I'm leaving off new communities for this month, but feel free to pop any in the comments you'd like to share!
So, I'll begin at the 0.19.11 update for Lemmy. I'd planned in some time for the update, and to begin with things went fairly normally, with usual server updates etc. Then it came time to update Lemmy itself to 0.19.11 - however, instead of the ~10 minutes of suggested downtime, it quickly became apparent that something wasn't quite right. The logs gave no indication of anything happening, and after an hour it became apparent that something hadn't worked. With not much to go on, and all the various logs showing that absolutely nothing had apparently happened, I cancelled the upgrade process and tweaked the logs so I'd get a bit more information than the standard logs usually show. This turned out to be an internal DNS issue, where Lemmy wasn't talking to the database.
So, onwards again I fixed that issue and reapplied the update, to which it had appeared to start working. Excitedly, I give it ten minutes as advised, to come back and see that it was still going.. and going.. and going.. and two hours later it was saying something was still happening. Well, by this point I'd run out of time and trusted in the process that it would resolve itself. I had unfortunately had to go to sleep as I had work the next morning, and a baby to look after.
By the next morning, still nothing. However, I can't SSH in to the server on my phone (nor would I ever want to in case it was stolen etc) and all the backend stuff is behind layers of protection, so the site was down for the day while I had to do real world work nonsense.
By the time I'd got home, it still thought it was ongoing, so I made the decision that likely something was broken and cancelled the update. Of course, letting something go to town for hours on the database meant it was probably ruined, or would require completely investigating - ain't nobody got time for that. So I crossed my fingers and hoped my backup solution had worked which it did with flying colours. I restored the database and the site came back to life.
Of course, I don't like running old versions, so I rescheduled which went much much smoother the second time around, taking 8 minutes. Whew.
For a few days, everything looked great. Metrics looked spot on and the site was running like it always was, and then bam - the server completely locked up:
A restart did eventually fix this, and things returned to normal, but not quite the same.
On the 24th April, I started getting reports of the site returning various 502 and 504 errors, and also issues with images loading. Broken images has been a feature of the image proxy for a while, but this was on another scale, with sometimes whole pages of broken images and a connection timeout on every refresh. I was, in hindsight, very out of my depth here.
I started trying to see what the cause of the timeouts where, and it became apparent quickly that the site was being hit by previously-unseen levels of traffic:
On the above graph, the left hand side was the usual amount of traffic we'd get, and on the right was the new levels we were receiving. This was likely causing the server to effectively be DDOS'd, and uncovering issues with the performance of the server. For a comparison of the usual amount of traffic we get, here's a graph from the last server update:
Many configs were tweaked. Much database performance monitoring took place. Things were turned off and back on indiscriminately. Unfortunately not much worked.
It was at this point our new Admin, Gazby, joined the team - and quickly got to work diagnosing and fixing so many of the issues.
Some of the changes include much better backups, reduced latency in image storage (the images moved from the USA to EU storage), images being served from i.lemmy.zip to allow for better caching to reduce the load on the server, complete review and tweak of all the configs and server setup, and a detailed list of things we need to work on going forwards (plus no doubt lots of things I am forgetting).
This has really helped to stabilize the site, and while we can see it's not 100% perfect, the amount of 502/504 errors should have massively reduced, and the site should be almost as it used to be.
One of things Gazby has also worked on is reporting and insights into the server, and so for example here's a graph showing the 504 errors today:
We've got some plans on the horizon we're working on which should increase the performance of the site, which possibly/probably will include a server move to rule out hardware issues or latent config issues we can't find.
Whatever triggered these issues I am still not sure - it looks like it could be scrapers/AI bots hammering the site, and so we've put some measures in place here. If you use the old front end then you may have noticed a cloudflare challenge to try and prevent the server being overloaded - also a prewarning, the old front end is no longer maintained, and unless someone steps in, it is not compatible with the next version of Lemmy. Therefore at that point it will be retired unless someone works on it to bring it back in line. It is also the cause of a LOT of the server traffic problems, so it probably isn't too much of a bad thing. Maybe someone will rewrite it to be better :)
As it stands, things still aren't perfect - we had an issue where the bit of the server that actually directs you to the right place got overwhelmed, and so we put a fix in for that, but they are hopefully a lot better than they were a couple of months ago.
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.
I know you've come for the shiny pictures, so here you go!
CPU over last 30 days:
RAM over last 30 days:
Disk space used:
Here's a few new ones for you! Lemmy DB size:
Images database size:
Here's our current actual images stored:
Here's the cloudflare overview:
Here's requests:
Bandwidth:
And here is visitors:
And finally, traffic by country (mostly federation traffic remember!)
So hopefully that fills everyone in on where we're up to, and what we're working on, but if you have any questions please ask away below!
One final thing - on the 10th June, Lemmy.zip turns two years old!! 🥳 🎉 I'm hoping to do something nice for it, similar to last years (which is here if you haven't seen it!) - but a quick thank you to everyone who has been part of the ride so far!
Thanks
Demigodrick
Hello all,
It's that time of the month again!
Unfortunately we didn't have chance to do an April Fools this year (besides, reality is weird enough in itself) - as mentioned at the end of the last update I was expecting a child, who has safely arrived and is amazing, whilst also ruining my sleep schedule and eating all of my free time!
I am sneaking a few minutes in though, to write this as they sleep!
We've had loads of new communities this month, so I'll highlight certain ones to keep the list a bit more manageable:
In terms of actual lemmy.zip development, things have been a little quieter. We updated to 0.19.10 when it was released - Changelog here, and we've implemented a couple of ways of trying to filter out Nicole PMs. If anything, its pushed me to learn new ways of interacting with Lemmy to harden the moderation support capabilities of Zippy. Before 0.19.10 was released, you may have noticed a PM in your inbox with something along the lines of "Removed by Admin" if Zippy picked up Nicole spam - with the introduction of 0.19.10, PMs are deleted when we ban an account, so you won't need to see that anymore.
And that's it really - Sami and Druid have done a fantastic job keeping on top of applications and mod reports, and hopefully everything will continue to run smoothly :)
I will schedule some small maintenance downtime when I get a chance, but I'll give plenty of notice.
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I am sure that long term readers are bored of reading this now, but it is really is true that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads. If you’ve gone to the trouble of creating an account, then come and be part of the Fediverse! Please don’t be a lurker! :)
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.
It's all you're really here for anyway ;)
CPU:
RAM:
Disk space:
Image storage:
Cloudflare overview:
Requests:
Bandwidth:
Unique visitors:
Traffic:
Apologies, bit of a shorter one than usual, but I am severely sleep deprived! (and will moan about it at every opportunity)
As always please let me know if there is anything you'd like to see me include in these, otherwise I'll be back next month with another one!
Thanks
Demigodrick
Hello all,
That time of the (shorter) month again!
We've got a couple of new communities for you:
A few things happened this month.
Firstly, the smaller things - We updated to Lemmy 0.19.9 which was quick and easy. We also migrated the image database from Sled to Postgres which was relatively easy once it got going, but was not quick. But that piece of work is done now, and should result in tiny performance improvements now we have so many images.
We also implemented the Geoblock of UK IP addresses (more info here). This took a couple of goes to get it how I wanted it to work, but the UI is now blocked to UK IP addresses, and federation traffic is still allowed to and from the UK.
We've also implemented something I talked about at the end of the last server update, which I am calling "Default Blocks" (for lack of a better name) for new accounts. Basically, when a new account is created, we apply a couple of instance blocks for instances that we get the most complaints about - so far, it's Hexbear (who have recovered their domain if you're interested) and lemmygrad. Lemmy.ml was a candidate too, but there are a lot of bigger communities there and we dont really get complaints/reports from our users about that instance.
The messages a new user gets when they've signed up to Lemmy.zip, and the welcome post it links to, have been updated to reflect this, and give instructions on how to remove these blocks (in the user profile -> blocks) in the same way you'd add/remove any other block (because that's all we're doing in the background). The updated new user post is here, and we're open to feedback if anyone can think of ways to add or tweak it to make it easier for new users.
This way, I think, is the best of both worlds when we're trying to weigh up keeping new people on Lemmy compared against defederation. Obviously we're not an instance that goes wild on defederating because we want people to see the content they want to see, but this can turn some newer people away who likely would've stuck round had the onboarding experience been a little easier and clearer. With default Lemmy, there isn't really an onboarding experience - many instances use the default lemmy experience which is an email to let you know your account has been approved, if they've enabled this, otherwise you're kind of left wondering. One of the things we saw a lot of was after an application was created, the user would try to immediately log in and then reset their password, not realising (nor having read the application text!) that they would have to wait.
This is why we've put so much effort into things like a welcome email, welcome PM linking to various places such as new users posts, how to change the site theme, how to get help if you need it etc, and deny emails if we don't accept an account with the reason why and how to challenge that decision, to name a few.
This also doesn't effect existing users who signed up before we put this into testing, it's something Zippy does to the account as part of the signup process, so it won't change anything for long term accounts - It's just new accounts that will have the blocks applied.
In time, we might extend this if there are more extreme instances popping up. We might also review it for communities too, although that functionality doesn't exist in Zippy yet.
The ultimate aim is to provide a friendlier landing page for new users and an easier transition from other similar sites to Lemmy, without cutting everyone off from that content if they want to partake in it which is what Defederation would do. Defederation would also prevent users from the other instances in taking part in our content, which many do in a fair and rule-following way, and that wouldn't really be something we want to do either. So this is the fairest way we can do this, while still giving the user the choice to remove those blocks at any time as and when they wish to.
As always, happy to answer any questions in the comments.
Finally, I'm trialing out a Mastodon server at mastodon.zip (or click here for a direct invite). At the moment its entirely separate from Lemmy.zip but if it gets used we'll likely bring it under the same banner at some point.
If you’re new to Lemmy.zip - WELCOME! I hope you’re enjoying your time here :)
I am sure that long term readers are bored of reading this now, but it is really is true that the easiest way you can support Lemmy.zip is to actively engage with content here: upvoting content you enjoy, sharing your thoughts through posts and comments, sparking meaningful discussions, or even creating new communities that resonate with your interests.
It’s natural to see fluctuations in user activity over time, and we’ve seen this over the wider lemmy-verse for some time now. However, if you’ve found a home here and love this space, now is the perfect opportunity to help us thrive.
If you want to support us in a different way than financially, then actively interacting with the instance helps us out loads. If you’ve gone to the trouble of creating an account, then come and be part of the Fediverse! Please don’t be a lurker! :)
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.
The bit you all really want to see. All over 30 days unless otherwise stated.
CPU:
RAM:
File storage:
Image storage (lifetime):
Cloudflate stats - overall:
Requests:
Bandwidth:
Unique visitors:
Country traffic:
So that's it for this month - if there is anything else you'd like to see I'm happy to share it if I can.
Next month's update may be late or missing entirely, and I may not be around as much for the next couple of months - I am expecting my first child around March/April and I think I might have slightly less time than I do now for a little bit! :)
The site is in good hands with Sami and Druid, so please do reach out to them or in the matrix chat if you need support with anything.
Thanks,
Demigodrick
Hey, welcome! :)