7
submitted 1 week ago* (last edited 1 week ago) by Dave@lemmy.nz to c/support@lemmy.nz

As many of you will know, we have been borrowing server hosting, which kept costs very minimal. Many of you donated to the host (thanks!), but a while back they stopped taking donations as they wound down operations including transferring mastodon.nz to new people and gave me notice that we would eventually need to move to other hosting.

That was quite some time back, and now I've been notified that the time has finally come. At some point in this month we will swap over to a new host. It will also involve significantly increased costs to run the server (up from almost nothing), and most likely we will move to a VPS off of the current shared but dedicated hardware to more appropriate specs, and move to using object storage for images. All this will likely require some down time, which I will try to give notice of. I'm currently planning for this to be quite significant (perhaps done over the course of a whole weekend), since this would involve migrating all images to object storage which must be done with pict-rs shut down (others have reported 8+ hours for this), as well as transferring all other data to a new VPS, then a slow period of time as it catches up on federation. The second day would be a backup in case something went wrong.

When this move happens, I'll also start to do regular financial updates. In the past people have been interested in running costs, but they have been minimal out of pocket due to the server hosting being free to me. With the increased costs it makes sense to start regular updates about what it costs to run the instance, similar to what other instances do. There is a very wide range in what this could cost, and it's not clear right now what the lowest spec server is that we could get away with. But I am hoping we can keep this under $50 a month. People have asked previously about donations, a little while back I set up a page so you could donate directly to Lemmy.nz, here is the post about it (with the emphasis on not donating if it will materially impact you): https://lemmy.nz/post/21494039

Before moving forward, I'd like some input from Lemmy.nz users on some specific questions:

Currently we are hosted in NZ, how important is it to continue this? (with cost being the main barrier)

We will try to host in NZ if we can, but if cost becomes an issue, is it a big deal to host in Australia or elsewhere? We will want to try to stay close to NZ so NZ based users have low latency

Do people object to Cloudflare?

Currently we use Cloudflare, with benefits around loading faster for people in other countries (due to the CDN) and the ability to more easily handle AI scrapers.

Cloudflare puts us in the power of a large company and reduces our independence, but it also reduces bandwidth usage and have tools to more easily fight AI scrapers that cripple the server. Cloudflare offers captcha services that help this.

Remember, users on other instances that don't proxy images will also be loading images directly from our servers, which Cloudflare will reduce the latency of due to their CDN.

If people object to Cloudflare, I am willing to attempt to avoid using it, adding Anubus and managing IP blocklists, but it will be more work. It will also be a slightly higher cost, as we will need a higher traffic allocation since Cloudflare currently caches a lot of image traffic.

Pātai?

If you have any questions, let me know!

top 24 comments
sorted by: hot top new old
[-] piwakawakas@lemmy.nz 3 points 1 week ago

Hey Dave,

Sounds like a big job, I'd offer to help, but I honestly have no idea what I'm doing.

In terms of support, I'm happy to contribute a small amount monthly on an ongoing basis.

For server hosting, NZ is probably ideal, but I'm probably going to be happy with anywhere that doesn't currently have an authoritarian leaning government. I know that cuts down options significantly.

Cloudflare, I'm not a huge fan of, although I know that they keep a large part of the net safe. They have issues of their own. I saw recently that the Canadian instances were switching to a Canadian product - deflect - again I don't really know what I'm doing so I don't know if it's feasible or not.

https://lemmy.ca/post/59882785

I'm not sure exactly where I sit in terms of the Lemmy population, but I'm relatively left leaning and privacy focused compared to the general population.

[-] terraborra@lemmy.nz 3 points 1 week ago

In addition to no authoritarian leaning jurisdictions, I also wouldn’t want the server to be based in a country that has age verification laws that apply to hosting. Yes I have a vpn, but the point is I shouldn’t have to use it just to access what used to be freely available information.

I stopped donating after we found out that we’d need to shift hosts but happy to start again.

[-] piwakawakas@lemmy.nz 2 points 1 week ago

That's a fair point actually. I saw there was a lot of outrage about discord.

There has been talk of implementing similar laws here though, unfortunately.

[-] Dave@lemmy.nz 1 points 1 week ago

I also wouldn’t want the server to be based in a country that has age verification laws that apply to hosting

Errr... that's a good point about avoiding Australia, I hadn't thought of the possible ramifications of that.

But also... let's hope our own version doesn't go through: https://www.legislation.govt.nz/bill/member/2025/0216/latest/whole.html

[-] BlueEther@no.lastname.nz 2 points 1 week ago

Hi Dave, good luck.

When I was looking at where to host No Last Name, I decided on Australia for three reasons, 1/ Cost 2/closeness in location 3/closeness in laws.

On CF, as much as I hate them I use them, so would support keeping on using them

How much traffic do you see at the moment and what storage amounts? Object storage for images should be a saving on cost. NZD50 probably wont get you far on shore.

[-] Dave@lemmy.nz 1 points 1 week ago

Traffic is a tricky one as there's a significant amount of traffic just for backups, and the backups contain a lot of images (the entire proxy cache as it's not easy to separate that out). It's incremental but there's a big churn in proxied images. On object storage I guess you don't backup the same way, but have history in the object storage?

Everything is on hard drives at the moment, perhaps 150GB images and another 40 for the main database, plus some extra for bits and pieces. But that will change with images in object storage.

Zappie has $25USD NZ VPSs with 6vCPUs and 4GB that you can add more RAM to at like $2/GB/mo or something. That's the cheapest NZ host I've found, and I don't mind paying a little extra to be in NZ.

What are the specs of your server?

[-] BlueEther@no.lastname.nz 2 points 1 week ago

I was using a QUBIT-4 from https://quantumcore.com.au/linux-vps, I picked that up for 30% off at some stage. When SWMBO said I was spending too much on domains/hosting/offsite backup/et I had to rationalise some of my spending and moved to OCI (yes I know, worse than almost anything else). Their free tier is really good for specs - I'm only paying for the off site backup of the DB/configs etc

[-] Dave@lemmy.nz 1 points 1 week ago

Ah yep. That QUBIT-4 seems pretty decent. How much RAM does your instance use these days? That's my biggest concern, as it's hard to tell what the minimum requirements are when the OS and db think unused RAM is wasted RAM and just gobble it up whatever is available for a disk cache.

[-] BlueEther@no.lastname.nz 2 points 1 week ago* (last edited 1 week ago)

sitting at about 5G reserved for pict-rs/postgres/lemmy. Virt use is using all 23 of the 24G of cause.

If you can swallow, and have no self worth, then Oracle is a fantastic deal:

  • 4vCPU
  • 24G ram
  • 50G ssd
  • 200G Block storage.

[-] Dave@lemmy.nz 1 points 1 week ago

That is amazing for a free server! I remember applying years back and getting rejected lol.

But yeah, I wouldn't use it for Lemmy.nz. I'm still hopeful about NZ hosting.

[-] Dave@lemmy.nz 1 points 1 week ago

Sounds like a big job, I’d offer to help, but I honestly have no idea what I’m doing.

Now you mention it, losing this host is effectively losing my backup. Is there a Lemmy.nz regular that does know what they are doing?

In terms of support, I’m happy to contribute a small amount monthly on an ongoing basis.

Thanks!

For server hosting, NZ is probably ideal, but I’m probably going to be happy with anywhere that doesn’t currently have an authoritarian leaning government. I know that cuts down options significantly.

It seems even just jumping across to Aussie hosting gives a good price drop, but it does seem feasible that we could do NZ hosting. The cheapest I have found so far is zappiehost.com who have NZ servers (and actually happens to be where the hardware we are on is sitting at the moment), but happy to hear other options if anyone knows a good and inexpensive NZ host.

Cloudflare, I’m not a huge fan of, although I know that they keep a large part of the net safe. They have issues of their own. I saw recently that the Canadian instances were switching to a Canadian product - deflect - again I don’t really know what I’m doing so I don’t know if it’s feasible or not.

I think Lemmy.ca ended up registering as a not-for-profit which gets them free use of that Deflect service. I'm not convinced it's important enough to pay for, given out of the box settings don't usually work as at least for Cloudflare, any bot blocking kills federation so it needs to be carefully managed anyway.

[-] MadPsyentist@lemmy.nz 2 points 4 days ago

I know enough to help out. Developer for the past 10 years. I know my way around a network happy to help out if you need a rubber duck to bounce ideas off of

[-] Dave@lemmy.nz 1 points 4 days ago

Nice, thanks for the offer. Do you happen to use Matrix?

[-] MadPsyentist@lemmy.nz 2 points 4 days ago

Nope. Mostly a signal dude myself. However i did just make a matrix account on matrix.org

[-] Dave@lemmy.nz 1 points 4 days ago

Yeah not many people use tend to use Matrix, however, for Lemmy admin stuff it's prevalent. For spam coordination, Matrix. For Lemmy support or Pict-rs (Lemmy image backend) support, there are various matrix rooms. Mbin and Piefed stuff also happens on Matrix. I guess the federated nature of Matrix is attractive to Fediverse admins.

If you want, you could join the Lemmy.nz matrix room (linked from this post), or just message me directly on Matrix (my account should be linked from my profile page - though I don't seem able to see the button myself, it should be there for you). Then we can chat 🙂

[-] deadbeef79000@lemmy.nz 2 points 1 week ago

Assuming a CF style CDN offload, what kind of capacity does that instance need?

I'm entirely ignorant of the architecture of a Lemmy instance. Is it a monolithic daemon or is a a bunch of semi-independant services that can be distributed over several hosts?

[-] Dave@lemmy.nz 1 points 1 week ago

It is horizontally scalable to some extent, but really what were talking about with Cloudflare is a distributed cache to many nodes around the world. It would sit in front of lemmy not be a part of it (much like CF does). If we decide we don't want Cloudflare, we wouldn't run our own CDN. Instead we would just accept that people far away would have a bit of a laggy experience. It's probably not that bad since most apps will be loading things before you get there anyway, and we aren't doing real time things.

[-] deadbeef79000@lemmy.nz 2 points 1 week ago

Sorry, poorly worded now I reread what I wrote.

I meant that, assuming a CDN, what are the CPU, RAM, and filesystem requirements for our Lemmy instance itself? And, can the Lemmy software stack be distributed over multiple hosts? Sounds like some of it can if there's is an image daemon of some sort that can be backed by an object store.

Also I remember you talking about federation/ActivityPub being really sensitive to network latency a while ago. Having the instance hosted closer to others network-wise might be useful.

All that's leading towards some cloud provider in the APAC region. but then again AWS, Azure, GCP, and Oracle are all fucking evil so it's be better to not depend on those.

[-] Dave@lemmy.nz 1 points 1 week ago

I meant that, assuming a CDN, what are the CPU, RAM, and filesystem requirements for our Lemmy instance itself?

This is a question I have been trying to get answered and haven't made much progress! I know the current system we are on is more powerful than would be needed, but I haven't been able to get any info on what minimum requirements are. I guess because of all the unknowns, it kind of depends. I also think that with or without a CDN the specs probably don't change, as serving static files isn't that hard for a server. It's the bandwidth that you get capped on which would need to be higher without Cloudflare. And this assumes we can effectively mitigate AI scrapers, which are a big problem in bursts - but I don't want to mitigate them by getting bigger hardware 😓.

And, can the Lemmy software stack be distributed over multiple hosts? Sounds like some of it can if there’s is an image daemon of some sort that can be backed by an object store.

Lemmy has some parts that could be put on multiple hosts, but it's unlikely that would be needed instead of just bumping the specs a little. Mostly you can solve bottlenecks by running parallel containers. I am hoping a VPS with 4 vCPUs and 4-6GB RAM will be a good starting point. We may end up needing 8GB RAM though (currently we have 16).

Also I remember you talking about federation/ActivityPub being really sensitive to network latency a while ago. Having the instance hosted closer to others network-wise might be useful.

For a long time, federation was sequential in that an instance, say Lemmy.world, would send one upvote or one post, then wait for a success response before sending the next one. This was important to ensure things arrived in the right order, e.g. if you edit your comment twice, you need to make sure the right version is what appears on other instances. Lemmy.world grew so big they were creating federation activities faster than they could send them to us, so someone else had made batching software that I had running on a VPS in Finland to specifically handle Lemmy.world (directing lemmy.nz traffic to the VPS with the help of the Lemmy.world admin team). I had it running for maybe 6-9 months, but this was solved perhaps around 12 months ago when Lemmy got the ability to send federation activities in parallel, so it's no longer an issue.

All that’s leading towards some cloud provider in the APAC region. but then again AWS, Azure, GCP, and Oracle are all fucking evil so it’s be better to not depend on those.

None of those have made the shortlist haha

[-] BaconWrappedEnigma@lemmy.nz 2 points 1 week ago* (last edited 1 week ago)

1. Use Cloudflare

Cloudflare has the power to be evil but that haven't shown that streak yet. The biggest risk IMHO is that, in the most useful configurations, most of the traffic passes in cleartext. So, if Cloudflare were compromised by hackers or a nation state, then passwords and emails and private messages could all be slurped up. I won't speculate which kind of nation state would be powerful enough to compromise Cloudflare.

Cloudflare R2 is pretty cheap for serving images (we just pay for storage, not outgoing bandwidth). BlueÆther mentioned Object Store. I would look at R2 first.

2. Consider Self-Hosting

I don't know what kind of load lemmy.nz has been under but I would imagine you could run the services using docker compose and still meet the demand. The hardware likely doesn't matter too much as long as it has enough RAM and an SSD. The most complex part about this is making sure the backup and restore strategy works. Mostly, that the restore strategy works. I have an Intel NUC on a cheap Eaton UPS that broadcasts my renegade signal to the world. You can run this using Cloudflare Tunnels so that you don't have to open a port on your router or expose your IP address. It helps to treat the server as if it were on a hostile network. So, you want to lock it up into its own VLAN or network segment to prevent a server compromise from pawing through your draws.

Postgres has a lot of different ways to keep "hot backups" with one of the more popular ones being WAL. Rclone with BackBlaze and a USB hard drive will get you pretty far on the backups.

I'd be happy to help set it up.

3. Consider Seeking Sponsorship

A local hosting company like SiteHost or Catalyst might offer to donate a VPS in exchange for "Thank You" in the footer.

If you are trying to gauge the potential for community financial support then I feel like you'd find enough people willing to support the hosting to make it viable.

[-] Dave@lemmy.nz 1 points 1 week ago

For 1: Thanks, good to know there are also people happy with Cloudflare.

I already have experience and accounts with Backblaze for object storage, that I use for Lemmy and personal backups. That also seems to be cheaper than Cloudflare as well. So unless a nice local provider shows up for a similar cost, or people have strong objections, I'll probably just go with that. I think the free egress of 3x data stored should go a long way to covering our need.

2: So Lemmy.nz is currently on dedicated hardware, but that hardware is managed by someone else and I have a Proxmox LXC container allocated that I fully control. Lemmy is all set up through docker compose currently - I think even the largest servers are set up like this since docker is the supported method for install, though the large instances have horizontally scaled federation.

In terms of hosting at home - it's a bad idea, for the sole reason that content is federated so you don't have control over it, and you really want to have control over what is in your house, sitting on your hard drives.

I already handle backups with a local (to the server) and remote copy stored on B2. Lemmy has instructions for exporting the database, which can be done live. So that's all sorted.

3: Hey if anyone knows someone then we could consider it, but it would be nice to just kinda do our own thing.

[-] BaconWrappedEnigma@lemmy.nz 2 points 1 week ago

you really want to have control over what is in your house

I hadn't considered this. That's kinda scary.

Another option is a legal structure that can have a bank account and pay for infra. I know they just made Incorporated Societies a bit more complex to manage. Even an Ltd with a clear constitution would work.

[-] Dave@lemmy.nz 1 points 1 week ago

Yeah at the moment I just see it as something I'm funding with some help from others as they choose to chip in. If it grows to be a problem, we can look at doing something like lemmy.ca did with FediCan, but they are a magnitude of order larger than us so it seems a bit preemptive.

[-] BaconWrappedEnigma@lemmy.nz 2 points 6 days ago

Maybe we could broker a deal between the different national bodies to form Fedi-CANZUK

this post was submitted on 09 Feb 2026
7 points (100.0% liked)

Lemmy.nz Support / Meta

382 readers
1 users here now

You can ask questions about the Lemmy.nz instance here.

This community also serves as our Meta community, with posts about updates to the Lemmy software on Lemmy.nz or other things relevant to Lemmy.nz users and not relevant to other instances.

founded 2 years ago
MODERATORS