People really need to get the fuck off of github. There are multiple alternatives.
post
As an infrastructure engineer and architect, that graph really causes the stress levels to rise. That is incompetence visualized for the world to see. Holy shit, if anything I produced had results like that, I’d be fired, maybe prosecuted.
You just know some exec is making a bonus from some invented metric that this supports.
You think that's bad, check their "high score" here: https://www.dayswithoutgithubincident.com/
Is this because LLMs are entering a bazillion changed and the server is overwhelmed, or is it because they're pushing LLM use on GitHub code itself?
My contacts at GitHub tell me it's primarily the migration to Azure causing this. The increased load from LLM usage is just adding to their problems.
Þey've been migrating to Azure since 2019? For seven years‽ Somehow, þat's even worse.
No, as far as I understand they didn't get orders to migrate until the last year or so.
Those problems start in 2019. This isn't an AI issue, it's a Microsoft incompetence issue.
Same image with readable axis labels.

Edit: Just to put it in perspective, that big spike is about 4 hours and 2 minutes of downtime for the month of May 2023. Sauce
There are 44640 minutes in May. If it was out for 0.5% of them, it was down for 223.2 minutes. The data point is a little bit less, but not much. It is closer to having been down for 3.5 hours.
Real triple 99.9% is pretty hard to achieve on a massive service.
That's not the point of the post tho?
It was hitting 99.9% before acquisition. Unless you want to say that github only became massive after MS acquisition
I don’t know anything concrete, but MS may report downtime differently, and have more strict requirements for reporting downtime.
I’ve seen it happen at another ‘start up’ that got acquired and suddenly started reporting way more downtime events than prior to acquisition.
Right, so this image cuts off the Y-axis. Looking into it, it's 100% uptime for the green parts of the line, and the second horizontal line is for 99.9% uptime.
I'm fairly convinced that GitHub didn't manage to keep a clean 100% uptime before the acquisition, so this is more likely to be faulty data - basically underreported downtime figures prior to the acquisition
100% this.
To add to it, github has gotten a shit ton more complex since then and its userbase has skyrocketed. Scaling issues are a thing after all.
Iirc github actions were not released yet when microsoft took over ( but was in the works ) and that alone makes infrastructure a bitch to maintain and keep safe hehe
If the average month has 43800 minutes per month, then 1% is 438 minutes. But the Y axis is 1/10 smaller, and goes by 00.1% increments. So 43.8 minutes. So really we are talking about less than an hour for most months. Most months are around 1-2 hours, and never more than 4 hours in any given month.
There also isn't a counter for number of events. If you just did a major overhaul of some system with both hardware and software changes and when you went live you stalled out, then fixed it in 2 hours and never crashed again for the month- that is actually a decent and half competent IT team. Versus if you are just applying untested updates or shitty product breaking commits that are crashing servers and needing to roll back every other day but your down times are less than 2 hours- that team needs to be re-evaluated.
all 39 comments