We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

top 50 comments

sorted by: hot top controversial new old
[–] 78 points 3 years ago (1 child)

Thank you for the transparancy!

  • source
  • hideshow 2 child comments
  • [–] 38 points 3 years ago

    Thanks for all your hard work @ruud@lemmy.world. It's not easy!

  • source
  • [–] 17 points 3 years ago (1 child)

    is that users can’t login anymore

    Ouch, that's bad :(

    Thank you for your work, as always :)

  • source
  • hideshow 2 child comments
  • [–] 14 points 3 years ago (4 children)

    Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!

    Dessalines:

    Nutomic:

  • source
  • hideshow 4 child comments
  • load more comments (4 replies)
    [+] 14 points 3 years ago (3 children)
  • load more comments (2 replies)
    [–] 14 points 3 years ago

    UI: 0.18.0 BE: 0.18.1-rc.1 πŸ‘

    seems to be working for me, I was already logged on

  • source
  • [–] 13 points 3 years ago (8 children)

    Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?

  • source
  • hideshow 14 child comments
  • [–] 6 points 3 years ago* (2 children)

    Federation is completely borked with .18 servers. Its very difficult for us to interact with https://lemmy.ca in any way: subscribing, upvoting, comments, posting... its all bugged.

    Its maybe not that big a deal because Lemmy.world "has the most users", so in some regards its https://lemmy.ca's loss but... we need to restore reliable federation... especially before the July 1st rush IMO.

    The .17 to .18 upgrade is basically a soft-defederation event, because of whatever this bug is between the two versions.

  • source
  • parent
  • hideshow 2 child comments
  • load more comments (2 replies)
  • load more comments (2 replies)
    [–] 7 points 3 years ago (1 child)

    As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.

    We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.

  • source
  • hideshow 1 child comment
  • load more comments (1 reply)
    [–] 7 points 3 years ago

    Oh. Good luck with the next try!

  • source
  • [–] 6 points 3 years ago* (2 children)

    Thanks for the transparency. Maybe it’s a good idea to have a test instance and some test cases/validation done there before updating the main instance. This is a regular process in any software/tech company/stack.

    Testing should never be done directly on the prod instance.

  • source
  • hideshow 2 child comments
  • load more comments (2 replies)
    [–] 6 points 3 years ago (1 child)

    @ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?

    With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.

  • source
  • hideshow 2 child comments
  • [–] 6 points 3 years ago* (last edited 3 years ago) (6 children)

    Eveyrthing (UI) is extremely slow now. Not an improvement at all.

  • source
  • hideshow 6 child comments
  • load more comments (6 replies)
    [–] 4 points 3 years ago

    Seems 0.18 is up... and...

    OH NO ! The interface changed ! Bring it back ! Bring it back... /s

    Just kidding... It's looking good, so it's time to explore a little and find all the new stuff.

    Nice work @Rudd, thanks for all the hard work.

  • source
  • [–] 4 points 3 years ago

    Seems to be working fine!

  • source
  • [–] 4 points 3 years ago

    Thanks for the info. You're doing a great job!

  • source
  • [–] 4 points 3 years ago (1 child)
    load more comments (1 reply)
    [–] 4 points 3 years ago

    My current session works now with 0.18.0 + 0.18.1.rc1

  • source
  • [–] 4 points 3 years ago (2 children)

    I am glad y'all rolled back. I had just enabled 2FA and could not login to my account. Rolling back removed 2FA so I was able to log back in. Thanks.

  • source
  • hideshow 2 child comments
  • load more comments (2 replies)
    [–] 4 points 3 years ago (4 children)

    Jerboa crashes constantly for me now, doesn't even let me view anything. JSYK!

  • source
  • hideshow 4 child comments
  • load more comments (4 replies)
    [–] 4 points 3 years ago (9 children)

    Jerboa is almost unusable for me at the moment. A lot of crashes, pages not loading, messages not sending etc. But I know that's inevitable at times. Restarting my phone seemed to make it worse. But I know you guys are on it! πŸ‘

  • source
  • hideshow 9 child comments
  • load more comments (9 replies)
    [–] 3 points 3 years ago

    PSA: If you've been using lemmy.world during the upgrades and rollbacks your browser might've cashed some stuff that quickly got out of date. If you have problems (especially JSON errors), try doing a hard refresh (usually Ctrl + F5) on the site. It might log you out.

  • source
  • [–] 3 points 3 years ago

    Linux/Firefox After upgrade many things didn't work, but they continued not to work after rollback.

    Turns out I needed to clear the cache, then everything worked fine again.

  • source
  • [–] 3 points 3 years ago

    Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.

    Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though

    any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?

  • source
  • [–] 3 points 3 years ago (4 children)

    Was worth a try! But yeah that was pretty bad 😬

  • source
  • hideshow 4 child comments
  • load more comments (4 replies)
    [–] 3 points 3 years ago

    Running one of the largest and most vibrant Lemmy instances certainly makes this deployment a great test case for the devs and the system overall. @ruud@lemmy.world this community really appreciates you!

  • source
  • [–] 3 points 3 years ago (1 child)

    I'm so new, I'm a baby, I have no idea what I'm doing. I did just create my account, I tried using the app but it wouldn't let me. Saying server version too low. I'm sorry for my ignorance, but is this what that means? And this means we can't use the app yet right?

    Also, thank you for all of this. I will be here for it all, I'm staying forever! I already love this so much more than anything else πŸ₯°

  • source
  • hideshow 1 child comment
  • load more comments (1 reply)
    [–] 3 points 3 years ago

    Working fine from Lift-off!

  • source
  • [–] 3 points 3 years ago*

    Great! I'm getting a bug, when I upvote a comment or post, the "loading animation" takes forever to complete. If I refresh the upvote is there. It reminds me to the bug we had with comments and new posts.

    Edit: Is the same for comments.

  • source
  • load more comments
    view more: next β€Ί