Red for Four Days Straight
A B2B kids' fashion seller's order portal has an automated gate that scans the code before every deploy. This gate went red four days in a row. Every day an "issue found" alert popped up.
Digging into the cause was a bit surprising. Three weeks earlier, in a security fix, we had deliberately removed the plaintext passwords that had been written literally into the code. If a password sits right there in the code, anyone can steal it. Removing it was correct.
But the gate was still holding an old standard. It was checking that "those three password constants must exist in the code" to pass. Security had improved, yet the checker ruled it "broken."
The fix was actually simple. We flipped the standard. Now it passes only when plaintext passwords are *absent*.
Let Me Retell It as a Building Guard
Here's an easier analogy. An apartment building upgraded its entrance security from keys to fingerprint scanning. Much safer.
But the guard's checklist still had an old item: "Confirm 3 keys in the key box." Since it switched to fingerprints, there are of course no keys. So the guard reported "issue found" every single day.
Having no keys isn't a problem. It's a good thing. What was outdated was the checklist. Our gate was exactly this guard.
The Neighbor Went Red Too
The same day, another service's checker was also red for four days. This time a small flaw remained in some code added as a trial of a new feature. And the data-saving jobs that run automatically every day kept inheriting that flaw and falling over with it.
The data did nothing wrong. The pipe carrying the data was sick. It's a shape you often see when a red light lasts for days.
The two cases looked completely different on the surface: one after a security fix, one after adding a new feature. But the root was identical: the standard the checker looks at couldn't keep up with the changes the actual code had walked through.
So, Four Lessons
One. Checkers age just like code. When what you're protecting evolves, the check standard has to evolve too. Yesterday's right answer can become today's wrong one.
Two. "Build passed" and "all gates passed" are different. On my machine I only watched the build, but the gate was running other checks before it. Don't trust that everything's fine off a single green light.
Three. If red lasts four days, the culprit is usually not "today's commit." For a failure that runs on for days, the cause is usually planted long ago. Staring only at what you changed today keeps you barking up the wrong tree.
Four. When a gate fails, add one more question. Don't only ask "Is the code bad?" — also ask "Is the gate old?" The more automation grows, this one question saves a lot of time.
One Line for Today
When a checker goes red, before you suspect the code, ask the checker's age first.