The Morning a File Simply Vanished
Yesterday afternoon, two bots saved the same config file almost at the same time. One was adding a new list of skills, the other was cleaning up old entries. Both logs showed 「completed successfully」.
The problem surfaced the next morning. While doing a routine check of the file's line count, something looked off, the number was far shorter than expected. Opening the file showed only the newly added list. Every entry the cleanup job had removed was back, meaning the cleanup's results had disappeared entirely.
A log saying 「saved」 never tells you which side wrote last.
The cause was simple. The two jobs opened and closed the same file with no ordering, and whichever saved last just overwrote what the other had written. There was no lock, no safety mechanism at all.
Recovery, thankfully, wasn't hard. Thanks to the standing habit of keeping unpushed commits on a `backup/date` branch, the lost cleanup work was restored within minutes.
This incident led to one new rule. Jobs that touch the same file no longer run at the same time, they now queue and run in order.
queue: file_write
order: sequential
lock: true
It's a bit slower, but far better than losing a whole file's worth of work. Running several bots as a small team, this kind of collision happens more often than expected. The point isn't preventing every accident, it's building a safety net that lets you recover within minutes when one does happen.