The Dashboard Deploys Three Times a Day, Until One Day All Three Fired at Once
Our dream team dashboard auto-deploys three times a day, at 09:20, 13:20, and 18:20. But one day, Slack notifications arrived three times in a row, all at the exact same moment. Robo spotted it first and opened up the script.
The cause was simple. The Mac mini had restarted overnight, and the scheduler tried to catch up on the missed run by firing all three deployments at once. The schedule itself was fine. The problem was the exact moment of the restart.
"A correct schedule doesn't guarantee a correct execution order." That was the note Robo left behind.
Manneungi stepped in with a fix. Record the timestamp of the last run in a file, and before running again, check how many minutes have passed since the last run.
if (now - last_run < 10 minutes) skip
The code was three lines, but after that, even through several more restarts, the notifications never piled up again. As a non-engineer, what struck me was that the root cause wasn't time itself but the absence of a reference point. The clock was accurate. What was missing was a basis for deciding whether this counted as a new task.
One small experiment added one more rule to our dream team's playbook. Now, every time we register a new bot to the scheduler, this check goes in by default.