So today I discovered that there's a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn't live inside any source control. This morning it entered a terminal state, and because it overwrites its state there's no way to revert it.
I'm currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
What's extra frustrating is the previous guy did create a git repo of these types of hacks, but this one doesn't live in it for no discernible reason.
Job security
He does charge a consulting fee to "fix" these issues
Almost all of them are dumb shit like this, where something is built in super hacky and dumbass ways.
It’s his kill switch and he forgot to check in.