The Alert That Rang Thirty Times a Day
Our Dream Team bots posted a Slack message every time they finished something. Sync done, backup done, article generated. That came to about 30 alerts a day. The trouble was that the failure alerts that actually mattered got buried in between. Last month we found 4 failure alerts only after a full day had passed.
Two Changes
First, success alerts are no longer sent right away. They are collected and sent once, at 6 PM. Second, failure alerts still go out immediately, but with a tag at the very front so they stand out at a glance.
[SUCCESS-BUNDLE] 18:00 | Done today: 27
[FAIL] Article generation | one-line cause
Failure alerts are limited to a one-line cause. If it runs long, people stop reading it again.
Two-Week Comparison
• Alerts per day: from 30 to around 4
• Delay before a failure was noticed: from 3 hours on average to 20 minutes
• Missed failures: from 4 a week to 0
An Unexpected Discovery
With fewer alerts, we ended up trusting the bots more. Quiet now means things are running well. Still, a day with no news at all made us uneasy, so even when the completed count is zero, the evening bundle always sends 「Today: 0」. Silence and a stalled bot are two different things.
Today's Note
More alerts do not make you feel safer. Fewer alerts get read.
The smaller the team, the more expensive human attention becomes. Teaching the bots to speak less is part of the design.