Skip to main content

Alert Rules Configuration

Overview​

Alert rules determine which incoming alerts from your monitoring systems trigger investigations. A rule is a per-cluster match filter. Any alert that matches a rule is investigated automatically; there is no separate enable/disable or auto-investigate toggle.

Setup​

1. Navigate to Alert Rules​

In the OpsWorker portal, go to Configurations -> Alert Rules.

2. Create a Rule​

Click Create Rule and configure the match filter:

FieldDescriptionExample
NameA descriptive name"Critical production alerts"
Namespace (namespaceName)Regex matching Kubernetes namespacesprod-.*
Severity (severityKey + severityFilter)Label key to read for severity, plus an RE2 regex to match its valuekey severity, filter `critical
Label (labelKey + labelFilter)Arbitrary alert label key, plus an RE2 regex to match its valuekey team, filter payments

Rules apply per cluster. Filters use Go RE2 regular expression syntax.

3. Save​

Save the rule. From then on, matching alerts trigger investigations automatically.

4. Verify​

To confirm the rule works:

  • Wait for a real alert that matches your filter criteria, or
  • Send a test alert from your monitoring system

Example Rules​

Investigate Critical Alerts in Production​

  • Namespace: prod-.*
  • Severity: key severity, filter critical

Investigate Payment Service Alerts​

  • Label: key service, filter payment-api
  • Severity: key severity, filter critical|warning

Combining Filters​

Filters within a rule are combined with AND logic. For example:

  • Namespace prod-.* AND severity critical
  • This matches only critical alerts in production namespaces.

Create multiple rules for OR logic (for example, one rule for critical alerts, another for payment service alerts).

Best Practices​

  • Start narrow: begin with critical alerts in production namespaces
  • Expand gradually: add rules for warning severity and additional namespaces as confidence grows
  • Review regularly: check the daily digest to see which alerts are being missed

Next Steps​