775
GitHub Outages Since Microslop Acquisition (thelemmy.club)
submitted 4 months ago by to c/technology@lemmy.zip
55 comments fedilink hide all child comments
 

top 50 comments

sorted by: hot top controversial new old
[–] 188 points 4 months ago (4 children)

  • permalink
  • fedilink
  • source
  • hideshow 8 child comments
    •
  • [–] 83 points 4 months ago* (3 children)

    I see two nines

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 35 points 4 months ago (2 children)

    I'm stupid what does zero nines uptime mean?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] [S] 65 points 4 months ago (2 children)

    These services measure their uptimes in number of nines, the more the better.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 48 points 4 months ago (2 children)

    Sometimes the humorous term "nine fives" (55.5555555%) is used to contrast with "five nines" (99.999%),[18][19][20] though this is not an actual goal….

    Maybe Microsoft misunderstood the assignment, and thought this was a goal. At their current rate, it’s certainly more achievable than the more traditional “five nines”.

    As an aside, I love how the following is preferences as “casual”, and then the author starts arguing semantics:

    Similarly, percentages ending in a 5 have conventional names, traditionally the number of nines, then "five", so 99.95% is "three nines five", abbreviated 3N5.[13][14] This is casually referred to as "three and a half nines",[15] but this is incorrect….

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 31 points 4 months ago*

    When contracting a service, usually there are clauses that specify that it needs to be fully working and available x% of time, and compensation may be due in case this goal isn't met.

    Let's say GitHub was down for 1 full day in the last year, that's 99.7% availability. That's "2 nines", but sometimes people might say "2 nines five", meaning "better than 99.5% uptime".

    I'd say that the expectation for a high availability service nowadays is "5 nines": 99.999% uptime. That's around 5 minutes of downtime in a full year. This kind of performance from a site like GitHub is just unacceptable...

  • permalink
  • fedilink
  • source
  • parent
    • [–] 58 points 4 months ago (1 child)

    Move slow and break shit

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • [–] 54 points 4 months ago (2 children)

    Obv a gross looking chart, but I am bothered that the left hand scale is trimmed off. I expect those are 10% increments, but wouldn't be shocked if Original was like 99.0, 98.0, 97.0, etc.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 50 points 4 months ago (1 child)

    You'd be surprised: https://damrnelson.github.io/github-historical-uptime/

    But weirdly enough it feels much worse using gh professionally than the scale makes it seem.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • [–] 46 points 4 months ago (3 children)

    I've worked on services with 5 nines of availability (i.e. 99.999% available, less than 5 minutes of downtime allowed per year). I've more frequently worked on ones with 4 nines, where you're allowed almost an hour of downtime per year. GitHub is now barely maintaining 2 nines. That's just embarrassing.

    Each "nine" you add is much more difficult. To get four nines you need people on call who can start working on a problem within 5 minutes and fix it within a few more minutes, and you can only get those calls once every couple of months. Five nines means that you need people at their desks in shifts ready to start fixing something the moment there's a problem because it would take too long for someone on-call to get their computer out, connect and authenticate. It requires warm backup systems that are sitting idle but ready to take over fully at a moment's notice.

    A two nines system is allowed to be down for 100x as long as a four nines system, and 1000x as long as a five nines system. It's almost 15 minutes of downtime allowed per day, compared to about 15 minutes every 3 months for a four-nines system. Gamers wouldn't even put up with a two-nines system for a video game. It's absurd to allow that for a critical piece of infrastructure for software.

  • permalink
  • fedilink
  • source
  • hideshow 6 child comments
    •
  • [–] 7 points 4 months ago* (1 child)

    Five nines means that you need people at their desks in shifts ready to start fixing something the moment there’s a problem

    No, it means you don't have outages. Ever.

    Five-nines is something like 7 minutes of downtime throughout the entire year. At best, you might have automated failover systems that require tiny outages. No human involvement, though, unless you're deal with some major breakage that would have killed the five-nines commitment that year, anyway.

    It's takes a human something like 5-10 minutes just to get out of bed and figure out the situation, anyway.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 14 points 4 months ago (6 children)

    No, it means you don't have outages. Ever.

    No, that's infinite nines, which isn't possible.

    Five-nines is something like 7 minutes of downtime throughout the entire year. At best, you might have automated failover systems that require tiny outages. No human involvement, though, unless you're deal with some major breakage that would have killed the five-nines commitment that year, anyway.

    Yes, you have automated failover systems. But, if something happens which causes those systems to fail over, you need to immediately investigate what happened and why. Even at four nines you have automatic failover, redundant system, hot spares, etc. But, you accept that sometimes not everything will work as planned and you'll need to fix something. Five nines is just that and more.

    It's takes a human something like 5-10 minutes just to get out of bed and figure out the situation, anyway.

    Right, which is why I said that four nines is your realistic maximum if you're going to have people on call who aren't actually at their desks. To get better than four nines you need to have around the clock coverage with people at their desks so when a system breaks you have eyes on it in something like 30s.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
    load more comments (6 replies)
    • [–] 28 points 4 months ago (1 child)

    Nothing to make a point like snipping off the y-axis scaling.

    I hate Microslop like any person with > 2 brain cells, but that graph is useless - all visible y-entries end in a 0 - might as well be 99.990, 99.980, 99.970, ...

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] 20 points 4 months ago (1 child)

    It's just Xitter's image viewer cropping it automatically; the original upload has it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 4 months ago (1 child)

    It is still bad practice to select a narrow window from a axis like this and show the difference that seems massive relative to what is shown but isn't that significant when we can see the relation to the whole.

    Graph 101

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 8 points 4 months ago (2 children)

    This is a commonly known issue with graphs and one that gets repeated without a lot of consideration for context. While it's generally a good basic rule to have graphs show the full vertical axis, it's not like it's a hard rule that needs to be followed 100% of the time. In this case for example, moving from 99.999% (five nines) to 99% (two nines) is a significant effect, it has importance. Displaying the full axis would make that difference unnoticeable and render the graph useless.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 3 months ago

    The difference between 97.1% uptime and 97.2% uptime is far less important than the difference between 99.98% uptime and 99.99% uptime, even though this graph shows the former as 10 times larger.

    You are right to complain that a graph that exaggerates uptime differences on the scale of 10%, i.e. showing the full linear range from 0% to 100%, would be useless. But by the same token, a graph that exaggerates uptime differences on the scale of 1%, i.e. the OOP, is also useless. That we happen to live in a world where github's uptime is varying at the scale of 1% doesn't make the scale any more useful.

    In this case, the graph of -log(1-uptime) would get you the "number of nines", which is commonly used because it's more insightful and more indicative of actual quality. Better still would be log(uptime(t)/(1-uptime(t)), which is functionally the same above two "nines", but also allows the plotting of low uptime services, such as individual seeders for torrents or specific nodes of a mesh, on the negative part of the scale.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 4 months ago

    Yes I absolutely agree but it has to be transparent and for me it is intentionally misleading to show it like this. Yes, it's still significant and still shows lack of care from microslop but context matters to me. Maybe more than to others :)) I acknowledge that I am special that way and this is fine for others

  • permalink
  • fedilink
  • source
  • parent
    • [–] 16 points 4 months ago (1 child)

    Surely they could just Copilot their way out of this mess lmao

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • [+] 12 points 4 months ago* (last edited 3 months ago) (1 child)
    [deleted]
  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • [–] 11 points 4 months ago*

    https://damrnelson.github.io/github-historical-uptime/

    A lot of this is GitHub Actions alone, but a lot of it isn't. I also don't know how well GitHub tracked outages before the Microsoft acquisition. It's entirely possible the graph looks so bad because they only took outage tracking seriously after being acquired. I don't know.

    Further related discussion on Hacker News

  • permalink
  • fedilink
  • source
    •
    [–] 9 points 4 months ago* (1 child)

    It is impressive how bad Microsoft is fumbling the bag

    Github has gotten extremely popular but it also sucks really bad

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • [–] 8 points 4 months ago

    Is that real? Because that... Makes it real clear...

  • permalink
  • fedilink
  • source
    •
    [–] 8 points 4 months ago (1 child)

    But the payment processing service has 9 nines of uptime......

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • [–] 3 points 4 months ago

    Remember when mSlop bought HotMail? Same shit, different decade.

  • permalink
  • fedilink
  • source
    •
    [–] 3 points 4 months ago

    How many of those outages were due to AI training?

  • permalink
  • fedilink
  • source
    •
    [–] 1 point 4 months ago (1 child)

    That's just fucking disgraceful.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    • load more comments
    view more: next ›