Alert Rules
What are Alert Rules
Alert rules control which incoming signals trigger automatic investigations. They act as filters between your monitoring systems and OpsWorker's investigation engine, so that only the alerts you care about are investigated.
Alert rules are configured per cluster. An alert that matches a rule on its cluster triggers an investigation; an alert that matches no rule is recorded but not investigated.
How Rules Work
When a signal arrives, OpsWorker evaluates it against the alert rules for that cluster:
flowchart LR
A[Signal arrives] --> B{Matches a rule on this cluster?}
B -->|Yes| E[Investigation starts]
B -->|No| D[Signal recorded only]
Rule Configuration
Each alert rule is scoped to a cluster and matches on the following fields. The severity and label matchers are key plus filter pairs, where the filter is a Go RE2 regular expression.
| Field | Description | Example |
|---|---|---|
| Name | A label for the rule | "Critical production alerts" |
| Namespace | RE2 regex matching the Kubernetes namespace | prod-.* (all production namespaces) |
| Severity key + filter | Label key to read severity from, plus an RE2 filter on its value | key severity, filter `critical |
| Label key + filter | Any alert label key, plus an RE2 filter on its value | key service, filter payment |
A rule matches when all of its configured fields match the incoming alert. There is no separate enable/disable or auto-investigate toggle; a matching rule triggers an investigation.
Examples
| Rule Name | Fields | Purpose |
|---|---|---|
| Critical production alerts | Namespace prod-.*, severity filter critical | Investigate all critical alerts in production |
| Payment service | Label service filter payment | Investigate all alerts for the payment service |
| Staging warnings and above | Namespace staging, severity filter `warning | critical` |
Best Practices
- Start narrow: begin with critical alerts in production, then expand as you gain confidence.
- Use namespace patterns: filter by namespace regex to cover groups of related services.
- Review regularly: use the daily digest and the dashboard to identify rules that should be added or adjusted.
Next Steps
- Configure Alert Rules - Step-by-step setup guide
- Create and Edit Rules - Manage rules in the portal
- Noise Reduction - Reduce alert fatigue