Skip to main content

Reduce Alert Noise

Scenario​

Your team receives 200+ alerts per week. Many are duplicates, low-priority, or non-actionable. Engineers are drowning in notifications and missing the alerts that actually matter.

How OpsWorker Helps​

Unified Visibility​

Alerts from AlertManager (Prometheus) and Datadog appear in one timeline in the OpsWorker portal. Grafana Alerting is supported when it sends alerts in AlertManager format; it is not a separate native source. Filter by severity, namespace, and cluster to understand your alert landscape.

Focused Investigation with AlertRules​

Per-cluster AlertRules are the main noise control that ships today. They filter incoming alerts by namespace, label, and severity (regex matching), and decide which alerts trigger an investigation:

  • Auto-investigate only critical alerts in production namespaces.
  • Record everything else for visibility without triggering investigations.

Basic Deduplication​

OpsWorker suppresses identical duplicate alerts: when two alerts share the same fingerprint and the same start second, the duplicate is dropped. This is exact-match dedup, not time-window suppression or silencing.

Daily Digest​

The daily digest (per-cluster, delivered to Slack at 09:00 UTC, also written to S3 and viewable in the portal) summarizes the last 24 hours with day-over-day (2-day) deltas. Use it to spot which namespaces generate the most alerts and adjust your AlertRules accordingly.

Roadmap: Alert Correlation​

Grouping multiple alerts from a single underlying issue into one incident is a planned feature, not available today. Each alert that passes your AlertRules currently spawns its own investigation; there is no incident entity or cross-alert correlation yet.

Outcome​

  • Focus on what matters: AlertRules scope investigations to actionable alerts only.
  • Fewer duplicates: identical alerts (same fingerprint, same start second) are suppressed.
  • Data-driven tuning: use the daily digest and portal visibility to refine your AlertRules.
  • Reduced fatigue: better signal-to-noise without over-promising correlation that does not exist yet.