2025-12-09:
Watchdog and cron
You already know I despise systemd a bit more than wayland, but the future will be in the future, and the past will still be in the past.
I have message queues and bees workers. Workers die each hour, but my lovely cron resurrects them and forces them to work.
It's just a single server and there won't be any more (but there will be fewer), so don't shame me, please.
There are a lot of workers (very few) and even more queues (a very few of few). They lived in harmony, but I did something wrong and the workers began their deadly loop and started eating memory. And they ate it all.
Using simple spells like "it worked before, which means my changes from yesterday are at fault" it was quite simple to find the broken worker. It was killed, and the rest were told to keep working while ignoring the corpse nearby.
And they worked. For an hour. And then they didn’t. Run them manually — works. Leave them alone — doesn’t.
That was weird, as cron wasn't touched, memory wasn’t leaking, but the workers didn’t want to work.
The book of life (/var/log/syslog | grep -i cron) told me that cron worked fine, but not since long ago. And then it didn’t.
It turned out that when the workers ate all the memory — watchdog killed cron. I doubt it realized that cron was the initial culprit; it was simply near the paw of death.
For the very first time in my life, watchdog killed cron! Unbelievable!
Would systemd timers survive? Would the paw of death smite them?
It's a mystery. I could run an experiment, but no. Imperative knowledge is only for people with weak faith.