Automatic Investigations

Overview

Automatic investigations run without any manual intervention. When an alert matches a configured alert rule with auto-investigation enabled, OpsWorker starts an investigation immediately, 24/7, regardless of who's on call or what time it is.

Investigation Trigger Types

Automatic is one of four ways an investigation can start:

Trigger	How it starts	Notes
Automatic	An incoming alert matches an alert rule with auto-investigation enabled	The focus of this page
Manual (free-text)	You type a problem statement and click Start investigation	Your description flows into the agents as user-provided context (see below)
Chat-initiated	Started from a free-form AI Chat session	Continues from the conversation context
Test	A synthetic alert used to validate setup	For configuration and demos

How It Works

sequenceDiagram
    participant Mon as Monitoring System
    participant OW as OpsWorker
    participant Rules as Alert Rules
    participant AI as Investigation Engine
    participant Slack as Slack

    Mon->>OW: Alert webhook
    OW->>Rules: Evaluate alert against rules
    Rules-->>OW: Match found (auto-investigate: on)
    OW->>AI: Start investigation
    AI->>AI: Discover, collect, analyze
    AI->>Slack: Post results

Configuration

Auto-investigation is controlled by alert rules:

1. Create an alert rule that matches the alerts you want investigated
2. Enable the Auto-investigate toggle on that rule
3. Any alert matching the rule will automatically trigger an investigation

Enabling Auto-Investigation

In the OpsWorker portal:

1. Go to Configurations → Alert Rules
2. Create a new rule or edit an existing one
3. Set your filter criteria (namespace, severity, labels)
4. Toggle Auto-investigate to Enabled
5. Save the rule

Manual Free-Text Investigations

You do not have to wait for an alert. From the portal you can start an investigation yourself by typing a free-text problem statement (for example, "checkout pods are restarting in production after the 14:00 deploy") and clicking Start investigation.

• Your problem statement is passed to the investigation graph as user-provided context (user_provided_context), so the agents factor your description into topology validation, runtime analysis, and the final synthesis.
• This is useful when you notice an issue before any alert fires, or when an alert does not capture the full picture.
• The same adaptive graph runs as for automatic investigations; only the trigger and the extra context differ.

Rate Limiting

OpsWorker prevents duplicate investigations for the same alert:

• If the same alert fires multiple times within a short window, only one investigation is created
• This prevents unnecessary work during alert storms
• Each unique alert combination (name + resource + namespace) is tracked

Coverage

Feature	Benefit
24/7 operation	Investigations run day and night, weekdays and weekends
No human trigger	The alert itself starts the investigation, no one needs to be awake
Consistent quality	Every alert gets the same thorough investigation process
Immediate start	Investigation begins the moment the alert arrives

Best Practices

• Start with critical alerts: enable auto-investigation for critical severity alerts first
• Expand gradually: add warning severity and more namespaces as you gain confidence in investigation quality
• Use feedback: rate investigation accuracy via Slack buttons to help improve results
• Review the daily digest: check the daily digest to see what's being investigated and what's being missed

Next Steps

• Configure Alert Rules: set up auto-investigation rules
• Investigation Lifecycle: what happens during an investigation