All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We'll see if that changes over the weekend...

you are viewing a single comment's thread
view the rest of the comments
[–] 156 points 2 years ago (3 children)

never do updates on a Friday.

  • source
  • hideshow 6 child comments
  • [+] 76 points 2 years ago* (last edited 2 years ago) (3 children)
  • [–] 42 points 2 years ago (2 children)

    And especially now the work week has slimmed down where no one works on Friday anymore

    Excuse me, what now? I didn't get that memo.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 15 points 2 years ago (1 child)

    Yeah it's great :-) 4 10hr shifts and every weekend is a 3 day weekend

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 2 years ago* (1 child)

    Is the 4x10 really worth the extra day off? Tbh I'm not sure it would work very well for me... I find just one 10-hour day to be kinda draining, so doing that 4 times a week every week feels like it might just cancel out any benefits of the extra day off.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 2 years ago

    I am very used to it so I don't find it draining. I tried 5x8 once and it felt more like working an extra day than getting more time in the afternoon. If that makes sense. I also start early around 7am, so I am only staying a little later than other people

  • source
  • parent
  • [+] 6 points 2 years ago* (last edited 2 years ago) (1 child)
  • [–] 2 points 2 years ago

    I changed jobs because the new management was all "if I can't look at your ass you don't work here" and I agreed.

    I now work remotely 100% and it's in the union contract with the 21vacation days and 9x9 compressed time and regular raises. The view out my home office window is partially obscured by a floofy cat and we both like it that way.

    I'd work here until I die.

  • source
  • parent
  • [–] 15 points 2 years ago (1 child)

    Yep, anything done on Friday can enter the world on a Monday.

    I don't really have any plans most weekends, but I sure as shit don't plan on spending it fixing Friday's fuckups.

  • source
  • parent
  • hideshow 2 child comments
  • And honestly, anything that can be done Monday is probably better done on Tuesday. Why start off your week by screwing stuff up?

    We have a team policy to never do externally facing updates on Fridays, and we generally avoid Mondays as well unless it's urgent. Here's roughly what each day is for:

    • Monday - urgent patches that were ready on Friday; everyone WFH
    • Tuesday - most releases; work in-office
    • Wed - fixing stuff we broke on Tuesday/planning the next release; work in-office
    • Thu - fixing stuff we broke on Tuesday, closing things out for the week; WFH
    • Fri - documentation, reviews, etc; WFH

    If things go sideways, we come in on Thu to straighten it out, but that almost never happens.

  • source
  • parent
  • [–] 4 points 2 years ago

    You posted this 14 hours ago, which would have made it 4:30 am in Austin, Texas where Cloudstrike is based. You may have felt the effect on Friday, but it's extremely likely that the person who made the change did it late on a Thursday.

  • source
  • parent
  • [+] -34 points 2 years ago (4 children)

    Never update unless something is broken.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 58 points 2 years ago (1 child)

    This is fine as long as you politely ask everyone on the Internet to slow down and stop exploiting new vulnerabilities.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 22 points 2 years ago (1 child)

    I think vulnerabilities found count as "something broken" and chap you replied to simply did not think that far ahead hahah

  • source
  • parent
  • hideshow 2 child comments
  • [–] 14 points 2 years ago (2 children)

    For real - A cyber security company should basically always be pushing out updates.

  • source
  • parent
  • hideshow 4 child comments
  • Exactly. You don't know what the vulnerabilities are, but the vendors pushing out updates typically do. So stay on top of updates to limit the attack surface.

    Major releases can wait, security updates should be pushed as soon as they can be proven to not break prod.

  • source
  • parent
  • [–] 7 points 2 years ago (1 child)
  • [–] 5 points 2 years ago (1 child)

    If it was Arch you'd update once every 15 minutes whether anything's broken or not.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 2 years ago (2 children)

    I use Tumbleweed, so I only get updates once/day, twice if something explodes. I used to use Arch, so my update cycle has lengthened from 1-2x/day to 1-2x/week, which is so much better.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 2 points 2 years ago (1 child)

    I really like the tumbleweed method, seems like the best compromise between arch and debian style updates.

  • source
  • parent
  • hideshow 2 child comments
  • I think a lot of what (open)SUSE does is pretty solid. For example, microOS is a fantastic compromise between a stable base and a rolling userspace, and I think a lot of people would do well to switch to it from Leap. I currently use Leap for my NAS, but I do plan to switch to microOS.

  • source
  • parent