this is what happens when you piss off elite hackers like musk and yudkowsky
this post was submitted on 14 Jan 2024
11 points (100.0% liked)
important instance shit
150 readers
1 users here now
founded 2 years ago
MODERATORS
And we will finally stop showing version numbers below every page here on awful.systems (waste of screen space & compute). Literally everyone knows lemmy never breaks during upgrades …
In today's episode of awful systems I found outt my phone's browser can't manage individual site cookies. Deleting them is an all-or-nothing deal.
Tfw the mono no aware hits you when you have to log out of everything just to make fun of techbros.
christ I’m sorry. my phone probably needs the same — the mother fucking cache-control immutable header some lemmy dev decided to hardcode into lemmy-ui’s app server seems to have triggered a bug in my phone where I’m both logged in and not simultaneously, and switching tab groups logs me out until I open a new tab
since all this unpleasantness is due to a bad cache-control header, hopefully it should all clear up when caches expire and pull in the new version of the site and corrected cache settings… in like 20 hours. fuck
They were using immutable cache control headers on resources that were, in fact, mutable? Uh oh.
oh yeah! it’s been causing a ridiculous number of problems for our instance and others (I thought beehaw was down for a long time but I’m realizing it was this cache issue) and basically makes upgrading to a backend that isn’t completely compatible with the previous backend impossible to do cleanly. and yet, I think I might have been the only one who’s discovered this issue, judging by the lack of relevant GitHub issues or other chatter about it.
as for why some lemmy dev chose immutable, all I can figure is they saw that rust is immutable (and I like rust a lot, but these are not good rust devs) and figured that meant immutability is a good thing everywhere regardless of the actual implementation. or maybe Google’s SEO tools insisted they needed it, and they didn’t read on or understand the versioned path requirement for cache busting? you’d think the devs of a federated platform would focus on literally anything other than SEO, but none of these folks had a plan.
anyway… only 7 hours remain until everyone’s browser hopefully marks the old content stale
I saw a few closed issues relating to cache control stuff. They've fixed some bugs related to it and also saw at least one where more aggressive cache control was implemented to improve performance.
oh hey speak of the devil https://github.com/LemmyNet/lemmy-ui/issues/2325 so this isn’t even purely a cache issue, somehow they also broke upgrading credentials between lemmy versions and nobody noticed
fucking brilliant. I might see if I can push a new version of the error page that doesn’t lie about what’s wrong now that caching hopefully isn’t busted
Lemmy-ui immutable static files have a cache busting hash in their path:
On awful.systems, I see you have replaced this hash with an undefined:
By the way, clean upgrades are completely possible if you know what you're doing. I frequently roll out new upgrades on lemm.ee without any downtime (so some servers running 0.19.1, some running 0.19.2 for example).
By the way, clean upgrades are completely possible if you know what you’re doing.
oh do tell
no thx but while we’re here
the reason why the cache busting broke is because I’m building for an immutable environment (NixOS sans docker) and my builds don’t have .git cause it’s a source of mutability. lemmy’s cache busting utterly breaks without git but it still incorrectly marks the files as immutable, breaking a ton of users. utterly broken behavior.
but the very funny part is the cache behavior isn’t even the bug that’s breaking everything right now. that’s this bug: https://github.com/LemmyNet/lemmy-ui/issues/2325 which you didn’t mention as a factor while you were trying your best to sound smart
also, have you ever gotten banned from an instance for being a shithead reply guy before? I’m betting this ain’t the first time
I frequently roll out new upgrades on lemm.ee without any downtime (so some servers running 0.19.1, some running 0.19.2 for example).
also motherfucker, those versions are compatible! no shit there’s no downtime for a hotfix upgrade. we got into this mess upgrading from 0.18.4 to 0.19.2 cause nobody seemed to think the upgrade process through in any detail
for anyone interested, the hack that NixOS uses to fix the asset path is here. the config we’re running predates that, and we may not be able to use that hack with a flakeified lemmy-ui (which is what makes it a hack — using src.rev like that is brittle and feels like it breaks Nix’s abstractions. maybe using git hashes like this is actually a bad fucking idea, huh? and the real fucked up part is, the UI knows its version so they could use that, but they woke up and chose jank)
Haha wow, now it's hitting me with the same thing I think. Keeps logging me out randomly and putting me in a state of Schrödinger's session cookie.
All hail the jank!
Edit: seems to happen if I refresh the page.
yep, exactly what I’m seeing except it stops when I open awful.systems in a new tab, but happens again if I switch tab groups. I don’t know how they managed this
Only two hard problems in CS: cache invalidation, naming things and off-by-one errors. But especially cache invalidation and especially especially when you make it even harder lol.
no such luck lol
This happened to me with mobile Firefox too, RIP my sessions
this is also in Chromium for desktop
Not sure if this is related, but any time I try to upvote or downvote a comment, I get this error:
Error: invalid_bot_actionhuh, I’ve never seen that before. let me look into this new jank (maybe your account somehow got marked as a bot account?)
fucking waaaaait is that why there’s a B next to your name? check your profile page and see if the “user is a bot” checkbox is enabled
[b]lessed [b]e the jank
I have used my immense technical skills nurtured over the course of decades to bypass the error page.
I am pleased to report that I have successfully cleared my cookies.
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
same
I’m impressed by the way some threads now have "-2" new comments. How does that work?
acausal trade
Looks like it reset 2fa on accounts, as well
checked the patch notes and that part is intentional. there was apparently bad 2fa UI before, which got people locked out, the reset is intended to rescue their accounts
Sharing just in case anyone else needs to see it - Safari on Mac gave me all of those login / logout things, and would not show any posts at all, just an error screen. I couldn't even see this note to know what the problem was, even when refreshing. I went to a clean Firefox browser, saw this, cleared Safari history and all is well.
Mac / OSX Sonoma.
I've deployed a fix for the JWT bug that's been breaking all our users, as well as the broken cache behavior. special thanks to the lemm.ee infrastructure admin/professional software developer/lemmy contributor below who bravely sacrificed their posting privileges by pointing out the source of the cache problem in the most jackass way possible and then insisting on a solution that doesn't work for our instance
there's one remaining lemmy-ui bug on my end I haven't been able to track down, and that's the weird half-logged-in session thing where depending on how you interact with the page, your session might disappear. that one stumps me to be honest; I'm going to try and debug with dev tools later, since currently it's only happening in Safari on my phone.
I humbly ask that you do not update the UI from 0.18.4, because pagination of comments is disabled? broken? with this update.
Alternatively, if you did update to 0.19.2 FE, and I still see the old functionality because of my cache+cookies, then the jank is even more powerful than I imagined.
the jank is extremely powerful, and it sounds like you’re running a mix of 0.18.4’s libraries and 0.19.2’s html and css right now. see elsewhere in this thread for the extremely drunk programming mistake some lemmy dev made that causes this
clearing your browser cache and cookies should get you a fully working lemmy UI, I hope (maybe try lemmy.ml first to see if pagination is working as expected on your end; they should be on the same version as us)
Nope, cleared cache and cookies for lemm.ee and the pagination for comments is still not working. Must be a bug with lemmy, pagination for posts works fine.
hmmm crap, I’ll keep an eye out for a hotfix. in the meantime, Proton might be a good option once it’s deployed if you don’t mind how it looks
Proton
Thanks for suggesting alternative frontends, proton doesn’t have what I wanted, but alexandrite does, and does it better than lemmy-ui.
this is wild, i can successfully log in and everything is fine in a chromium incognito window but not in a normal window even after clearing cookies and cache
EDIT: chrome://settings/content/all shows the cookie and I can delete it
firefox works after i just clear the cache
MOBILE SUGGESTION: if you have android, why aren't you using firefox with ublock origin already
lemmy on urbit on windows
Just in case anyone else runs into it: looks like this issue also completely wedged my install of Mlem - had to nuke the app from phone and reinstall, its sign-in procedure didn’t cope with this failure mode and it had no way to forget about account without also wanting to remotely delete an account (which I figure probably would’ve failed because no auth token, but I didn’t chance it)
I’ll log a bug their side when I have some headspace to write it up