Grafana alerting audit: 16 of 209 alert rules could never fire

Summary

A blind alert rule is one whose query returns zero series. In Grafana it looks healthy (health=ok, state=Normal) and is indistinguishable from a working rule, yet it can never fire.

On 22 August 2026 we checked every alert rule in an infrastructure of several hypervisors and dozens of virtual machines. Of 209 rules with an expression, 43 returned zero series. Of those, 27 were silent by design and 16 were broken: the metric source had died, moved or been renamed. Fifteen were fixed the same day, and the last one was removed on 22 September. Four rules — three rewritten and one new — are proven to fire.

Since 22 September the second half of the method runs in a daily watchdog: it tells "zero by design" from an orphaned rule, and when in doubt it calls a human. As of 14 October there are no unexplained blind rules among PromQL rules; log-based rules are still reported by the watchdog as query errors.

1. Hypothesis and method

The starting hypothesis: when nodes moved to a declarative OS (NixOS), some metric sources did not make the move, and the alerts built on them went silent.

The method has two steps:

1. For each of the 209 rules with an expression (PromQL or LogQL) we counted the series its query returns, through a proxy of the data source the rule is bound to. This way log-based rules are counted properly. The scheduled script that was already counting blind rules before the audit reports them as query errors.
2. For every zero we checked the base metric without thresholds and filters. Only this separates "zero by design" (absent() guards, thresholds over a live base, time windows) from "the source is dead".

Checking for series is not new in itself. Cloudflare's pint, for example, has a promql/series check that asks a live Prometheus whether a rule's series exist and explains why they don't. Here the rules live in Grafana, and we built the same check through data-source proxies; we have not checked whether pint can handle Grafana-managed rules. On top of the series check comes a second question: will the rule fire when the value really crosses the threshold (section 3)?

Fig. 1. How the audit separates "zero by design" from a broken rule.

2. What we found: 16 rules, three causes

The hypothesis held only halfway: 9 of the 16 rules were tied to the NixOS move, the rest broke for other reasons.

The lesson from the first row: a monitoring tunnel has two halves of configuration, one on the monitoring server and one on the node. A rebuild that updates one half and loses the other stops metric collection, and every rule over those metrics goes blind.

What we did:

• declared the node half in the nodes' declarative configuration, so the next rebuild will not lose it;
• updated the half on the monitoring server after verifying the nodes' new details over an independent channel, instead of trusting them on first connection;
• switched the log agent to container discovery and checked the labels in Loki;
• rewrote the rules that were watching old names and old locations.

3. Proof of firing

A zero on live data proves nothing: a rule that cannot fire also shows zero. So each of the four rules — three rewritten and one new — was evaluated twice through Grafana's rule evaluation API. The live value gives 0, a value pushed past the threshold gives 1. Only that pair proves the rule fires when the value crosses the threshold.

The new rule fires when a collector hangs. Two neighbouring rules take their data from that collector, and its freezing had been blinding them silently.

4. The daily watchdog: a classifier instead of a manual threshold

A scheduled script counts blind rules once a day; it was running before the audit as well. Until 22 September the alert "blind rules have grown" had a manual threshold: a measured number written into the rule. It went stale with every new absent() guard, because for such a rule zero series means health, so the blind count grows with normal work. The alert had been firing for months and was no longer read as a signal.

On 22 September a classifier was added to the script. It splits blindness into two kinds — not by the shape of the query, but by asking whether the metric exists at all:

• the metric exists but the query is empty: a filter, absent() or a selector matched nothing — that is health;
• the metric does not exist: the rule guards something that isn't there — that is an orphan.

Three decisions around the classifier:

• When in doubt, a rule counts as unexplained. A query the classifier cannot understand goes to a human. A classifier that silently vouches for a rule it doesn't understand is worse than none.
• Two-sided control. A separate set of 8 cases, positive ones included. A classifier that always answers "by design" would report zero unexplained rules and look perfect.
• No manual number. The alert fires if there is at least one unexplained rule. There is no baseline left to recalculate.

What did not work the first time: the first version of the classifier had three errors. It looked at the operators in a query and got it wrong where the comparison lives in a Grafana expression rather than in PromQL. Classification ran on a query string truncated for printing, so a comparison near its end was invisible. And label names from by (...) were taken for metric names. All three have been fixed.

After a day of alert review on 22 September, including the threshold change, 4 rules were firing instead of 11, and one of the four fires by design. On 14 October the watchdog learned to account for paused rules: they are reported on a separate line. As of that date there are no unexplained blind rules among PromQL rules, and 3 rules are paused. One of the three was paused that same day: its metric is written only by a test-scenario generator that is not currently running. Log-based rules are still reported by the watchdog as query errors.

5. The audit's own recommendation, disproved by measurement

The audit suggested removing a filter that excluded two nodes from one rule: the nodes now publish the metric, so the filter supposedly blinded the rule for nothing. Before making the change we measured it over 7 days. The nodes do publish the metric, but they were healthy only 71.5% of the time, with 102 and 120 transitions to "unhealthy". Without the filter the alert would have been firing all the time. The filter stays, and the measurement is recorded next to it.

The takeaway: an audit recommendation is also a hypothesis. It gets measured before it gets applied.

6. What this method cannot see

• Partial blindness. The series exist, but the rule sees fewer nodes than it should. A zero-series audit does not notice.
• A rule that is unsatisfiable by construction. An example from practice: while the condition being watched is absent, the metric exists and equals zero; when the condition occurs, the metric disappears, and "no data" is declared normal. No state of the world makes such a rule fire. In the healthy state the zero-series audit passes it, because the series exist. The two-value check misses it too: pushing a value past the threshold does not reproduce the metric disappearing. Only one question catches it: is there any value of the metric at which the rule fires, and does the metric actually arrive in that state?
• How this differs from the usual checks. Alert fatigue is visible right away: there are too many alerts. A blind rule is the opposite and invisible failure: the alert is missing when it is needed, and noise metrics will not show it. Rule unit tests (promtool test rules) run the expression on synthetic series, and health=ok only means the query executed; neither says whether the rule has live series. An always-firing heartbeat alert (a dead man's switch) catches the failure of the whole notification pipeline, not a single rule whose source has disappeared. The No Data state in Grafana helps only if the rule is configured to treat missing data as an alert; for rules where zero series is normal, that setting produces false alarms.
• A metric stuck in Pushgateway. A metric group that nobody has updated for a long time keeps serving series. To the audit its source looks alive, although no data has come from it for a long time.

Takeaways

• health=ok does not mean a rule can fire. Check the series count through the rule's data source and the base metric without thresholds.
• Proof of firing is a pair of evaluations: the live value gives 0, a value pushed past the threshold gives 1.
• When in doubt, the classifier hands a rule to a human, and it is itself checked by two-sided control: a zero on live data proves nothing.
• Audit recommendations, like any hypothesis, are measured before they are applied.