As many of you will know, we have been borrowing server hosting, which kept costs very minimal. Many of you donated to the host (thanks!), but a while back they stopped taking donations as they wound down operations including transferring mastodon.nz to new people and gave me notice that we would eventually need to move to other hosting.
That was quite some time back, and now I've been notified that the time has finally come. At some point in this month we will swap over to a new host. It will also involve significantly increased costs to run the server (up from almost nothing), and most likely we will move to a VPS off of the current shared but dedicated hardware to more appropriate specs, and move to using object storage for images. All this will likely require some down time, which I will try to give notice of. I'm currently planning for this to be quite significant (perhaps done over the course of a whole weekend), since this would involve migrating all images to object storage which must be done with pict-rs shut down (others have reported 8+ hours for this), as well as transferring all other data to a new VPS, then a slow period of time as it catches up on federation. The second day would be a backup in case something went wrong.
When this move happens, I'll also start to do regular financial updates. In the past people have been interested in running costs, but they have been minimal out of pocket due to the server hosting being free to me. With the increased costs it makes sense to start regular updates about what it costs to run the instance, similar to what other instances do. There is a very wide range in what this could cost, and it's not clear right now what the lowest spec server is that we could get away with. But I am hoping we can keep this under $50 a month. People have asked previously about donations, a little while back I set up a page so you could donate directly to Lemmy.nz, here is the post about it (with the emphasis on not donating if it will materially impact you): https://lemmy.nz/post/21494039
Before moving forward, I'd like some input from Lemmy.nz users on some specific questions:
Currently we are hosted in NZ, how important is it to continue this? (with cost being the main barrier)
We will try to host in NZ if we can, but if cost becomes an issue, is it a big deal to host in Australia or elsewhere? We will want to try to stay close to NZ so NZ based users have low latency
Do people object to Cloudflare?
Currently we use Cloudflare, with benefits around loading faster for people in other countries (due to the CDN) and the ability to more easily handle AI scrapers.
Cloudflare puts us in the power of a large company and reduces our independence, but it also reduces bandwidth usage and have tools to more easily fight AI scrapers that cripple the server. Cloudflare offers captcha services that help this.
Remember, users on other instances that don't proxy images will also be loading images directly from our servers, which Cloudflare will reduce the latency of due to their CDN.
If people object to Cloudflare, I am willing to attempt to avoid using it, adding Anubus and managing IP blocklists, but it will be more work. It will also be a slightly higher cost, as we will need a higher traffic allocation since Cloudflare currently caches a lot of image traffic.
Pātai?
If you have any questions, let me know!
Hey Dave,
Sounds like a big job, I'd offer to help, but I honestly have no idea what I'm doing.
In terms of support, I'm happy to contribute a small amount monthly on an ongoing basis.
For server hosting, NZ is probably ideal, but I'm probably going to be happy with anywhere that doesn't currently have an authoritarian leaning government. I know that cuts down options significantly.
Cloudflare, I'm not a huge fan of, although I know that they keep a large part of the net safe. They have issues of their own. I saw recently that the Canadian instances were switching to a Canadian product - deflect - again I don't really know what I'm doing so I don't know if it's feasible or not.
https://lemmy.ca/post/59882785
I'm not sure exactly where I sit in terms of the Lemmy population, but I'm relatively left leaning and privacy focused compared to the general population.
In addition to no authoritarian leaning jurisdictions, I also wouldn’t want the server to be based in a country that has age verification laws that apply to hosting. Yes I have a vpn, but the point is I shouldn’t have to use it just to access what used to be freely available information.
I stopped donating after we found out that we’d need to shift hosts but happy to start again.
That's a fair point actually. I saw there was a lot of outrage about discord.
There has been talk of implementing similar laws here though, unfortunately.
Errr... that's a good point about avoiding Australia, I hadn't thought of the possible ramifications of that.
But also... let's hope our own version doesn't go through: https://www.legislation.govt.nz/bill/member/2025/0216/latest/whole.html
Hi Dave, good luck.
When I was looking at where to host No Last Name, I decided on Australia for three reasons, 1/ Cost 2/closeness in location 3/closeness in laws.
On CF, as much as I hate them I use them, so would support keeping on using them
How much traffic do you see at the moment and what storage amounts? Object storage for images should be a saving on cost. NZD50 probably wont get you far on shore.
Traffic is a tricky one as there's a significant amount of traffic just for backups, and the backups contain a lot of images (the entire proxy cache as it's not easy to separate that out). It's incremental but there's a big churn in proxied images. On object storage I guess you don't backup the same way, but have history in the object storage?
Everything is on hard drives at the moment, perhaps 150GB images and another 40 for the main database, plus some extra for bits and pieces. But that will change with images in object storage.
Zappie has $25USD NZ VPSs with 6vCPUs and 4GB that you can add more RAM to at like $2/GB/mo or something. That's the cheapest NZ host I've found, and I don't mind paying a little extra to be in NZ.
What are the specs of your server?
I was using a QUBIT-4 from https://quantumcore.com.au/linux-vps, I picked that up for 30% off at some stage. When SWMBO said I was spending too much on domains/hosting/offsite backup/et I had to rationalise some of my spending and moved to OCI (yes I know, worse than almost anything else). Their free tier is really good for specs - I'm only paying for the off site backup of the DB/configs etc
Ah yep. That QUBIT-4 seems pretty decent. How much RAM does your instance use these days? That's my biggest concern, as it's hard to tell what the minimum requirements are when the OS and db think unused RAM is wasted RAM and just gobble it up whatever is available for a disk cache.
sitting at about 5G reserved for pict-rs/postgres/lemmy. Virt use is using all 23 of the 24G of cause.
If you can swallow, and have no self worth, then Oracle is a fantastic deal:
That is amazing for a free server! I remember applying years back and getting rejected lol.
But yeah, I wouldn't use it for Lemmy.nz. I'm still hopeful about NZ hosting.
Now you mention it, losing this host is effectively losing my backup. Is there a Lemmy.nz regular that does know what they are doing?
Thanks!
It seems even just jumping across to Aussie hosting gives a good price drop, but it does seem feasible that we could do NZ hosting. The cheapest I have found so far is zappiehost.com who have NZ servers (and actually happens to be where the hardware we are on is sitting at the moment), but happy to hear other options if anyone knows a good and inexpensive NZ host.
I think Lemmy.ca ended up registering as a not-for-profit which gets them free use of that Deflect service. I'm not convinced it's important enough to pay for, given out of the box settings don't usually work as at least for Cloudflare, any bot blocking kills federation so it needs to be carefully managed anyway.
I know enough to help out. Developer for the past 10 years. I know my way around a network happy to help out if you need a rubber duck to bounce ideas off of
Nice, thanks for the offer. Do you happen to use Matrix?
Nope. Mostly a signal dude myself. However i did just make a matrix account on matrix.org
Yeah not many people use tend to use Matrix, however, for Lemmy admin stuff it's prevalent. For spam coordination, Matrix. For Lemmy support or Pict-rs (Lemmy image backend) support, there are various matrix rooms. Mbin and Piefed stuff also happens on Matrix. I guess the federated nature of Matrix is attractive to Fediverse admins.
If you want, you could join the Lemmy.nz matrix room (linked from this post), or just message me directly on Matrix (my account should be linked from my profile page - though I don't seem able to see the button myself, it should be there for you). Then we can chat 🙂