I've recently told my boss I've been struggling and need extra help. I told him I was hiding it to avoid negatively impacting my career. He told me it was not a problem and suggested I take a week off work to refresh myself. I was shocked. My company is super helpful and is helping me with resources. I was shocked... I am shocked.

Have you told your work?

Timeline and reasoning behind recent infra changes

Recently, you may have noticed some planned outages and site issues. I've decided to scale down the size and resilience of the infrastructure. I want to explain why this is. The tl;dr; is cost.

Reasons

• I started discuss.online about 4 weeks ago. I had hoped that the reaction to Reddit's API changes would create a huge rush to something new, for the people, by the people; however, people did not respond this way.
• I built my Lemmy instance like any other enterprise software I have worked on. I planned for reliability and performance. This, of course, costs money. I wanted to be known as the poster child for how Lemmy should operate.
• As I built out the services from a single server instance to what it became the cost went up dramatically. I justified this assuming that the rush of traffic would provide enough donors to supplement the cost for better performance and reliability.
• The traffic load on discuss.online is less that extraordinary. I've decided that I've way over engineered the resilience and scale. Some SubReddits that had originally planned to stay closed decided to re-open. I no longer needed to be large.
• The pricing of the server had gotten way out of control. More than the cost of some of the largest instances in Lemmy while running a fraction of the user base.

Previous infrastructure

• Load balancer (2 Nodes @ $24/month total)
• Two front-end servers (2 Nodes @ $84/month total)
• Backend Server (1 Node @ $84/month total)
• Pictures server (1 Node @ $14/month total)
• Database (2 Nodes @ $240/month total)
• Object Storage ($5/month + Usage see: https://docs.digitalocean.com/products/spaces/details/pricing/)
• Extra Volume Storage ($10/month)
• wiki.discuss.online web node ($7/month)
• wiki.discuss.online database node ($15/month) [Total cost for Lemmy Alone: $483 + Usage]

Additionally:

• I run a server for log management that clears all lots after 14 days. This helps with finding issues. This has not changed. ($21/month)
• Mastdon server & DB ($42/$15/+storage ~ $60 total/month)
• Matrix server & DB ($42/$30/+storage ~ $75 total/month)

Total Monthly server cost out of pocket: ~$640/month.

The wiki, Mastodon, Matrix, & log servers all remained the same. The changes are for Lemmy only and will be the focus going forward.

First attempt

As you can see it was quite large. I've decided to scale way down. I attempted this on 7/12. However, I had some issues with configuration and database migration. That plan was abandoned. This is what it looked like:

Planned infrastructure

• Single instance server (1 Node @ $63/month total)
  • Includes front-end, backend, & pictures server.
• Database server (1 Node @ $60/month total)
• Object Storage ($5/month + Usage)
• Extra Volumes ($20 / month total)

[Total new cost: ~$150 + Usage]

Second attempt

I had discovered that the issues from the first attempt were caused by Lemmy's integration with Postgres. So I decided to take a second attempt. This is the current state:

Current infrastructure

• Single instance server (1 Node @ $63/month total)
  • Includes front-end, backend, & pictures server.
• Database server (1 Node @ $60/month total)
• Object Storage ($5/month + Usage)
• Extra Volumes ($20 / month total)
• wiki.discuss.online web node ($7/month)
• wiki.discuss.online database node ($15/month)

[Total new cost for Lemmy alone: ~$170 + Usage]

New total monthly server cost out of pocket: ~$330

My current monthly bill is already more than that from previous infrastructure @ $336.

Going forward

Going forward I plan to monitor performance and try to balance the benefits of a snappy instance with the cost it takes to get there. I am fully invested in growing this community. I plan to continue to financially contribute and have zero expectations to have everything covered; however, community interest is very important. I'm not going to overspend for a very small set of users.

If the growth of the instance continues or rapidly changes I'll start to scale back up.

I'm learning how to run a Lemmy server. I'll adjust to keep it going.

Here are my current priorities for this instance:

1. Security
   • This has to be number one for every instance. Where you decide to store your data is your choice again. You must be able to trust that your data is safe and bad actors cannot get it.
2. Resilience & backups
   • Like before, it's your data and I'm keeping it useable for you. I plan to keep it that way by providing disaster recovery steps and tools.
3. Performance
   • Performance is important to me mostly because it helps ensure trust. A site that responds well mans the admin cares.
4. Features
   • Lemmy is still very new and needs a lot of help. I plan to contribute to the core of Lemmy along with creating 3rd party tools to help grow the community. I've already began working on https://socialcare.dev/. I hope to help supplement some missing core features with this tool and allow others to gain from it in the process.
5. User engagement
   • User engagement would be #1; however, everything before this is what makes user engagement possible. People must be using this site for it to matter and for me to justify cost and time.

Conclusion

If you notice a huge drop in performance or more issues than normal please let me know ASAP. I'd rather spend a bit more for a better experience.

Thanks, Jason

There will be a short 5 to 10-minute outage as I scale down the environment. I was ambitious about the growth of this community during the Rexidus / Rexxit. The server sizes for discuss.online way outpace the daily load. Usage is typically well below 5% on all servers. Lemmy has become much more memory efficient, and I've made other adjustments to improve performance. Memory usage hangs around 10% for the backend and 30% for front-end servers.

The target is to reduce to better align with current donation payments. Server costs are over 10x the monthly donations right now.

Let me know if you have any issues. Please, remember to check https://status.discuss.online/ for updates during the outage.

Thanks, Jason

If you noticed some strange site behavior, it could have been caused by some load testing I performed.

A misconfiguration in rate limiting caused all users to share the same bucket. This has caused strange behaviors for some users.

I've stopped load testing and adjusted rate limits to prevent further disruptions.

Let me know if you still experience any issues.

You can always email: [email protected] if you have issues logging in.

Thanks, Jason

Hello ~~r/nerf~~ /c/nerf and welcome!

I'm the admin of this instance of Lemmy. I'm eager to grow Lemmy's presence on the web and this instance in particular.

Is there anything I can do to help anyone? What are some features you wish to see in Lemmy soon?

I recently did an AMA (which didn't get too much traffic) if you have any questions that might have already been answered: https://discuss.online/post/70663

Also, did you know there is a discuss.online matrix space? https://matrix.to/#/#community-channels:discuss.online

I'm happy to send registration codes to anyone wishing to join Matrix that doesn't already have an account. I'm happy to set up a c/nerf specific channel too!

I'm here to help.

Thanks, Jason

Are there any communities missing from Lemmy that you'd like to have started?

Or

Are there communities on federated instances you wish existed here?

Please let me know if you said yes to either of these questions! I'd love to get them started here. I can either help you do it, or I can start to build it up.

I'm hoping to grow this community, and part of doing that is as simple as asking you what you want.

Image by pch.vector on Freepik

We're currently running the latest release candidate of Lemmy 18.2. It has many fixes; however, there are still some remaining issues.

One example is that some buttons do not appear correctly on the dark themes. I will keep my custom dark theme as the default theme for now. I think it just looks better, even with broken upload buttons. I will attempt to apply a fix to it later today.

If you see any other issues, please, let me know so I can forward them to the core development team-- if you don't wish to do so yourself.

Thanks a lot! -jgrim

Why do you lurk?

I used to be a lurker when on Reddit. I'd post here and there but mostly just kept to myself. It was because something as large as Reddit felt like shouting into the void.

On Lemmy, the communities are small, and so are the instances. Is it lack of something to say, anxiety, or you're a bot? What's up!

I find myself being way more active on Lemmy & Mastodon than on Reddit & Twitter.

We've updated Lemmy 18.1. There may still be some issues; however, I believe it resolves much more than it creates. Let me know if you see anything funky.