@V0ldek @sailor_sega_saturn "That gibbering under the desk? Oh, that's just Azathoth. Poor thing got a look at the pump controller code last year. It's never been quite the same since."

From a lovely response to the Crowdstrike error and various speculation on what caused it (https://ruby.social/deck/@V0ldek@awful.systems/112824202708490681), comes this gem:

> all modern software lies upon a rotting pile of ancient mistakes.

To be clear: this is 100% true. As we slowly, painfully work our way toward being less awful at software engineering, we are better than we have ever been. As fucked as modern code is, old code was worse.

The lower in the stack you go, the more horrifying the revelations, just as a rule.

@V0ldek @sailor_sega_saturn have you read the writings of James Mickens, e.g. https://www.usenix.org/system/files/1311_05-08_mickens.pdf ?

@V0ldek @sailor_sega_saturn Thanks! You are talking straight from my heart!

I once helped one of my company's customers troubleshoot an issue that had seen the same ridiculous edge case error happen three times over the course of a few years. At one point the actual sustaining developer we worked with was able to narrow down a specific bit that was getting flipped somehow, and pitched that cosmic radiation was a plausible solution given how rarely this kind of thing impacted other customers.

It was at this point that we remembered that the customer was either a university with a nuclear physics lab or a hospital with a nuclear medicine program (can't remember now, ironically enough) that the server rack lived adjacent to.

some twenty four years ago i managed, amongst others, a company's samba and print server (that was at the time when all the company's servers were beige boxes with less memory and disk than the laptop i'm using to type this – and still they served a few hundred employees).

the machine developed a strange custom of hard-resetting itself, which we initially tracked to specific files being sent for printing; the behaviour was fully reproducible.

as it happened, it was a hardware fault somewhere between the mainboard and the integrated SCSI card; installing a separate SCSI card and reconnecting the disks and backup tape device fixed the problem. (i did not have the budget for a new serwer, no.)

establishing the actual cause took me fucking weeks.

the use of “DEI hire” is a shorthand for “i'm a massive racist shitweasel”

https://www.crowdstrike.com/blog/technical-details-on-todays-outage/

This is not related to null bytes contained within Channel File 291 or any other Channel File.

That to me implied that the channel file wasn't actually necessarily corrupt (or as corrupt as people thought), but that it triggered a logic error. In particular this point implies that it wasn't from garbage zero bytes in the file.

(That said I could have worded this better, in my defense I'm sick in bed and only half thinking straight)

go for the whole instance, the fucker is the admin there.