[thejefflarson]

What

A rule aggregates every matching series into one scalar (agg folds across time and series), so a firing alarm carried only {rule, metric, service} — never which node/pod crossed. For a cluster-wide rule like node load average (agg=max across all nodes) that left the operator — and fixer — to run kubectl top nodes by hand to find the hot box. This is the watcher-engine half of the cluster-repo TODO(JEF) on that rule.

How

When a rule fires, additionally resolve the one series that drove the breach — the series with the largest per-series aggregate for a gt rule, the smallest for lt — and attach its attributes JSONB (k8s.node.name, host.arch, …):

• webhook: a new optional attributes field (omitted on resolve and when unresolved), so fixer can act on the node, not just the rule.
• email: a Series: k8s.node.name=cluster-main, host.arch=arm64 line.

The attribution query (series_attrs_sql) mirrors eval_sql's window / predicate / rate / increase shape exactly but GROUPs BY (service, attributes) instead of folding across them, ordered in the breach direction, LIMIT 1. It runs only when a rule actually fires (not every breach tick) and is best-effort — no data or a query error yields no series and the page still goes out. No schema change (computed from the existing metrics.attributes column); injection-safe on the same terms as eval_sql (only the whitelisted agg_expr and the comparator-derived ASC/DESC are interpolated, every operand bound).

Tests

cargo clippy --all-targets -- -D warnings clean, cargo fmt applied, 25/25 alerts:: tests pass — new coverage: the attribution SQL shape (group-by, breach-direction order, limit 1, plain/rate/increase), render_attrs (sorted, bounded, skips empty/non-object), the email Series: line, and the webhook payload including/omitting attributes.

Follow-ups (separate repos)

• After this merges, watcher's CI rebuilds the image; the cluster repo's charts/watcher image digest must be bumped to deploy it, and the TODO(JEF) comment on the arm64 load rule removed.
• fixer can then surface the node from the alarm attributes in its diagnosis.

🤖 Generated with Claude Code

https://claude.ai/code/session_013cYVqzH7Xfwea7fAozdQK7