So today I discovered that there's a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn't live inside any source control. This morning it entered a terminal state, and because it overwrites its state there's no way to revert it.
I'm currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
What's a cron job?
Cron is a scheduler to run a program at a set frequency
The executive branch of the US government.
Scheduled job, but implies that it uses a cron format.
An older way of automating stuff.
It's not there by default nowadays, because systemd tends to fulfil their requirements.