Skip to main content

Automatic Investigations

Overview​

Automatic investigations run without any manual intervention. When an alert matches a configured alert rule with auto-investigation enabled, OpsWorker starts an investigation immediately — 24/7, regardless of who's on call or what time it is.

How It Works​

sequenceDiagram
    participant Mon as Monitoring System
    participant OW as OpsWorker
    participant Rules as Alert Rules
    participant AI as Investigation Engine
    participant Slack as Slack

    Mon->>OW: Alert webhook
    OW->>Rules: Evaluate alert against rules
    Rules-->>OW: Match found (auto-investigate: on)
    OW->>AI: Start investigation
    AI->>AI: Discover, collect, analyze
    AI->>Slack: Post results

Configuration​

Auto-investigation is controlled by alert rules:

  1. Create an alert rule that matches the alerts you want investigated
  2. Enable the Auto-investigate toggle on that rule
  3. Any alert matching the rule will automatically trigger an investigation

Enabling Auto-Investigation​

In the OpsWorker portal:

  1. Go to Configurations → Alert Rules
  2. Create a new rule or edit an existing one
  3. Set your filter criteria (namespace, severity, labels)
  4. Toggle Auto-investigate to Enabled
  5. Save the rule

Rate Limiting​

OpsWorker prevents duplicate investigations for the same alert:

  • If the same alert fires multiple times within a short window, only one investigation is created
  • This prevents unnecessary work during alert storms
  • Each unique alert combination (name + resource + namespace) is tracked

Coverage​

FeatureBenefit
24/7 operationInvestigations run day and night, weekdays and weekends
No human triggerThe alert itself starts the investigation — no one needs to be awake
Consistent qualityEvery alert gets the same thorough investigation process
Immediate startInvestigation begins the moment the alert arrives

Best Practices​

  • Start with critical alerts — Enable auto-investigation for critical severity alerts first
  • Expand gradually — Add warning severity and more namespaces as you gain confidence in investigation quality
  • Use feedback — Rate investigation accuracy via Slack buttons to help improve results
  • Review the daily digest — Check the daily digest to see what's being investigated and what's being missed

Next Steps​