The problem
A release went out missing a migration that another service needed. Nobody could enroll two-factor auth for a whole night, and we found out hours later. The error rate for a flow that few people use at night doesn't trip a generic alert, and by volume it looks like normal traffic.
The idea
Treat "a flow that used to produce events stopped producing them" as the signal, not just error rates. Three families of rules:
- Funnel-to-zero: for each critical step (sign-up, login, 2FA enrollment, withdrawals, transfers, deposits), compare the last few hours with the same hour-of-day slot on previous days. A step that normally has traffic and now has zero fires.
- Error ratio per endpoint: 12 auth and transaction endpoints, each 30-minute interval compared against the same hour of the previous day, with a minimum volume so a single request can't fire it.
- Known signatures: specific log lines that mean a flow is broken, grepped directly.
The build
A single Python script with no dependencies, authenticated against the Loki datasource with a Grafana service-account token. It runs once or in a loop every 30 minutes, keeps state in a local JSON file so the same alert isn't repeated within 6 hours, and posts to Slack through a webhook (or prints to stdout if the variable isn't set). About 70 instant queries per run, around 2 minutes. I chose a 3-day offset as the baseline after measuring that 7-day range queries took over a minute each; the detector has to be cheap enough to run forever.
Result
On its first production run it fired exactly once, on a known-signature rule, and the signal was real. The funnel-to-zero and error-ratio families produced zero false positives. A synthetic canary that exercises the enrollment flow on a schedule would have caught the broken release about 5.5 hours earlier than we actually found out.
What's next
Load the funnel and signature rules as Grafana alerts over Loki so detection doesn't depend on a script running on someone's machine, and add a synthetic canary that exercises the enrollment flow every 30 minutes so the baseline never depends on real users showing up at night.
What I'd tell another team
- "Zero events where there used to be some" is the alert that catches silent breakage. Error rates only catch loud ones.
- Compare against the same hour of day, not a rolling average. Traffic has a shape.
- Make the detector boring: no dependencies, local state, one file. It's the only way it survives.