8
submitted 4 months ago by cole@lemdro.id to c/lemdroid@lemdro.id

Sorry about that. Our hosting provider went down.

At the moment our architecture looks roughly like this:

  • Lemmy backed on a OneProvider VPS
  • everything else hosted at Fly.io (we heavily rely on the private networking that fly.io provides, they rock!)

In the future, I'm planning on moving Lemmy backend and pictrs to live in my basement. I have a 1 gig symmetric connection with a backup failover, so I honestly think it'll be more reliable than these cheap hosting providers and it will be cheaper.

Just want to give a quick update on what's happened and how we'll fix it going forward!

[-] cole@lemdro.id 29 points 5 months ago

nice to have that sort of freedom, since the majority of companies use some Microsoft products

8
Recent Slowdowns (lemdro.id)
submitted 8 months ago by cole@lemdro.id to c/lemdroid@lemdro.id

Hi! I know lemdro.id has been a bit slower than usual lately. I'm working to move us to totally dedicated hardware to mitigate this.

Expect early January, although I will make efforts to speed us up on our existing hosting provider.

[-] cole@lemdro.id 36 points 8 months ago

EULAs are not legally enforceable anyways

[-] cole@lemdro.id 37 points 8 months ago

nothing wrong with that.

you support the people you watch without having to see ads. also, comes with YT music.

it's a pretty good deal

[-] cole@lemdro.id 26 points 9 months ago

"privacy friendly" and "free cloud storage" are oxymorons. If you aren't paying for a product, you ARE the product

41
submitted 1 year ago* (last edited 1 year ago) by cole@lemdro.id to c/birding@lemmy.world

Captured these beauties near Olympic National Park

[-] cole@lemdro.id 27 points 1 year ago

right but unless you sign a contributor licensing agreement when you contribute then the copyright owner can't relicense code you contributed.

so if you contribute to a GPL codebase it's pretty legally perilous to try to unilaterally relicense code that isn't "yours".

this is pretty nebulous territory anyways, but I'd argue it's pretty unethical to relicense to a more restrictive license essentially "taking" the GPL code from contributors

[-] cole@lemdro.id 29 points 1 year ago

on the other hand, it is REALLY annoying

[-] cole@lemdro.id 27 points 1 year ago

that would literally defeat the entire purpose of the AUR

[-] cole@lemdro.id 27 points 1 year ago

source: https://xkcd.com/932/

(for those that want to read the alt-text)

11
submitted 1 year ago* (last edited 1 year ago) by cole@lemdro.id to c/lemdroid@lemdro.id

Retrieval of most pictures seems to not be currently working. I am still attempting to understand and resolve this.

Edit: This has been fixed

27
submitted 2 years ago by cole@lemdro.id to c/android@lemdro.id
14
submitted 2 years ago* (last edited 2 years ago) by cole@lemdro.id to c/lemdroid@lemdro.id

UPDATE DAY 2: backend has successfully been migrated onto new dedicated hosting after some pain. there should not be major downtime from here on. tomorrow I will be working on integrating a better backup solution and then I'll leave it alone for a little while.

UPDATE: I was able to deploy the database onto dedicated server hardware tonight, but have not finished moving over the other components I wanted to. You may notice a performance degradation due to increased database-backend latency (...or maybe it will just be better anyways, lol).

I will finish off work on this tomorrow!

Lemdro.id has been struggling with some performance issues lately, as you've likely noticed. This is due to changes made by our hosting provider that causes the database to run much slower. Tonight at 10pm PST, I will be putting lemdro.id into maintenance mode to migrate some parts of the infrastructure to a new dedicated server.

Thanks for your patience!

40
submitted 2 years ago* (last edited 2 years ago) by cole@lemdro.id to c/lemdroid@lemdro.id

Hey all! I've done a lot of database maintenance work lately and other things to make lemdro.id better. Wanted to give a quick update on what's up and ask for feedback.

For awhile, we were quite a ways behind lemmy.world federation (along with many other instances) due to a technical limitation in lemmy itself that is being worked on. I ended up writing a custom federation buffer that allowed us to process activities more consistently and am happy to say that we are fully caught up with LW and will not have that problem again!

Additionally, on the database side of things, I've setup barman in the cluster to allow for point of time backups. Basically, we can now restore the database to any arbitrary point in time. This is on top of periodic automatic backups which also gets pulled to storage both on my personal NAS as well as a Backblaze bucket (both encrypted of course).

Today, I deployed a new frontend at https://next.lemdro.id/. This one is very early stages and experimental but is being developed by https://lemm.ee/ and seems promising!

If you live outside of the US and experience consistently long load times I want to hear from you! I am deploying the first read replica node to Europe soon, so if you live in that region you'll soon notice near-instaneous loading of content. Very exciting!

Finally, looking for feedback. Is there anything you want to see changed? Please let me know!

[-] cole@lemdro.id 24 points 2 years ago

I feel like Dune was a good outlier to this. It's the only movie I've seen in theater in the last few years and I really enjoyed not having everything explained to me

109
submitted 2 years ago by cole@lemdro.id to c/android@lemdro.id

Google today announced a handful of wearable and navigation updates, starting with public transit directions in Google Maps for Wear OS.

71
submitted 2 years ago by cole@lemdro.id to c/android@lemdro.id
21
submitted 2 years ago* (last edited 2 years ago) by cole@lemdro.id to c/lemdroid@lemdro.id

I am rolling out the Photon UI as a replacement to the default lemmy UI right now. Initially, only about 50% of requests will be routed to Photon, determined by a hash of your IP address and user agent (sorry for any inconsistencies...). As I determine that this configuration is stable I will be slowly increasing the percentage until Photon is the new default frontend for lemdro.id.

If you have any difficulties, please reach out. Additionally, the "old" lemmy frontend will remain available at https://l.lemdro.id/

Edit: I am aware of some problems with l.lemdro.id. It wasn't designed to run on a subdomain so I'll need to add a proxy layer to it to redirect requests. A task for tomorrow!

FINAL EDIT: https://l.lemdro.id/ is now fully operational, if you choose to use the old lemmy UI it is available there

9
submitted 2 years ago by cole@lemdro.id to c/lemdroid@lemdro.id

Over the course of the last couple weeks, I managed to root cause and solve the problem causing stale sorting on lemdro.id. My apologies!

94
submitted 2 years ago by cole@lemdro.id to c/android@lemdro.id

We typically like Pixel phones a lot, but we have some reservations about Google's quality control

[-] cole@lemdro.id 39 points 2 years ago* (last edited 2 years ago)

Basically, the lemmy backend service for some reason marked every instance we federated with as inactive, which caused it to stop outbound federation with basically everyone. I have a few working theories on why, but not fully sure yet.

TL;DR lemmy bug, required manual database intervention to fix

This was a stressful start to a vacation!

For a more detailed working theory...

I've been doing a lot of infrastructure upgrades lately. Lemdro.id runs on a ton of containerized services that scale horizontally for each part of the stack globally and according to load. It's pretty cool. But my theory is that since the backend schedules inactive checking for 24 hours from when it starts that it simply was being restarted (upgraded) before it had a chance to check activity until it was too late.

theory:

  • scheduled task checks instances every 24 hours

  • I updated (restarted it) more than every 24 hours

  • it never had a chance to run the check

  • ???

  • failure

This isn't really a great design for inactivity checking and I'll be submitting a pull request to fix it.

[-] cole@lemdro.id 33 points 2 years ago

this might be an improvement

[-] cole@lemdro.id 87 points 3 years ago

Hello! Admin here at lemdro.id. This is the result of several problems in the default reference nginx config. I am working on resolving this right now and should have it fixed within the next 30 minutes!

view more: next ›

cole

0 post score
0 comment score
joined 3 years ago
MODERATOR OF