49
How do you handle cron jobs that silently stop working? (lemmy.world)
submitted 5 days ago by indidev@lemmy.world to c/programming@programming.dev
16 comments fedilink hide all child comments

Had a backup script running via cron for months. Worked fine until it didn't — turns out the disk filled up three weeks ago and the job started failing silently. Nobody noticed until we actually needed a restore.

The obvious answer is "check your logs" but let's be honest, nobody's reading cron logs daily for 15 different scheduled tasks across 4 servers.

What's your setup for making sure crons are actually completing? Do you just grep logs periodically, or do you have something more structured? Curious how others handle this without turning it into a whole project.

you are viewing a single comment's thread
view the rest of the comments
[-] kibiz0r@midwest.social 3 points 5 days ago

Can’t answer for them, but I can explain a similar situation I ran into:

Large external RAID I was using for telescoping backups with a restic cron job. Didn’t want to have it on all the time cuz it was noisy, so I put it on a smart plug and had it turn on and off as part of the job.

I really wanted a set of backup rules that worked like… “always a full snapshot of current state, plus as much of the following as will fit into the destination: daily for the past 7 days, weekly for the past 4 weeks, monthly for the rest until full” …but restic doesn’t (or didn’t) support that kind of dynamic rule set.

I tried to estimate it programmatically, but it never worked 100% and I frequently ran into out of space errors.

But of course, I never saw the actual disk usage in my day-to-day, because the RAID was only on at like 4 AM.

this post was submitted on 10 Jul 2026
49 points (96.2% liked)

Programming

27709 readers
591 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 3 years ago
MODERATORS
snowe@programming.dev
Ategon@programming.dev
UlrikHD@programming.dev
bugsmith@programming.dev
Spyro@programming.dev