Skip to main content

Investigation Flow

OpsWorker runs investigations as an adaptive graph of AI nodes, not a strictly linear pipeline. After topology and configuration validation, the graph branches: a critical alert-related configuration issue routes straight to final analysis (the config-remediation path), while everything else runs the conditional runtime investigation first (the runtime path).

End-to-End Data Flow​

sequenceDiagram
    participant Mon as Monitoring System
    participant API as API Gateway
    participant Norm as Normalizer
    participant Rules as Rule Engine
    participant AI as Investigation Graph
    participant SQS as SQS
    participant Agent as K8s Agent
    participant K8s as Kubernetes API
    participant DB as Data Store
    participant Slack as Slack

    Mon->>API: Alert webhook (POST)
    API->>Norm: Raw alert payload
    Norm->>DB: Store normalized signal
    Norm->>Rules: Evaluate against alert rules
    Rules->>AI: Match found, start investigation

    Note over AI: static_extraction
    AI->>AI: Regex field extraction (AI fallback if low confidence)

    Note over AI: topology_validation (one LLM node)
    AI->>SQS: BFS topology crawl within K8s call budget
    SQS->>Agent: Commands
    Agent->>K8s: get pod / svc / ingress / config
    K8s-->>Agent: Specs + status
    Agent->>SQS: Results
    SQS-->>AI: Resources + edges
    AI->>AI: Validate config, flag wiring issues, set has_critical_issues

    Note over AI: dependency_extraction
    AI->>AI: Service dependency graph (Pixie if enabled, else LLM)

    alt Critical alert-related config issue
        Note over AI: skip runtime, config-remediation path
    else No critical config issue
        Note over AI: investigation (conditional runtime, cycle budget)
        AI->>SQS: Get logs, events, live state
        Agent->>K8s: Multiple read queries
        SQS-->>AI: Runtime data (partial result if budget exhausted)
    end

    Note over AI: analysis (final synthesis)
    AI->>AI: Inject cluster/org memory + user context
    AI->>AI: Root cause + recommendations

    AI->>DB: Store investigation results
    DB->>Slack: Notification with summary

Nodes in Detail​

1. Alert Reception​

Monitoring system sends an alert via webhook to OpsWorker's API Gateway. Supported native formats: Prometheus AlertManager (also used by Grafana Alerting, which speaks the AlertManager webhook format) and Datadog. Any source that can post a webhook payload compatible with one of these formats works without custom configuration. Datadog is an alert source only; OpsWorker does not query Datadog during an investigation.

2. Normalization​

The alert is converted into OpsWorker's common format regardless of source. Key fields extracted: alert name, severity, labels, annotations.

3. Rule Evaluation​

The normalized signal is evaluated against configured alert rules. If a rule matches and auto-investigation is enabled, an investigation starts.

4. Field Extraction (static_extraction)​

Regex-based extraction identifies the affected namespace, pod name, and other context from the alert with no LLM call. If confidence is low (complex or non-standard alert formats), AI-based extraction is used as a fallback.

5. Topology and Configuration Validation (topology_validation)​

A single LLM node maps the dependency chain using breadth-first search within a bounded Kubernetes call budget AND validates configuration at the same time. A pod-crash investigation might discover Deployment, ReplicaSet, Pod, Service, Ingress. It produces resources plus edges, flags wiring issues (selector or port mismatches) and marks whether each is alert-related, and sets has_critical_issues.

6. Dependency Extraction (dependency_extraction)​

Builds a service-dependency graph for the affected workloads. Uses Pixie-observed traffic when Pixie is enabled, otherwise falls back to LLM inference. All cluster queries are read-only.

7. Runtime Investigation (investigation, conditional)​

Runs runtime analysis (logs, events, live state) only when there is no alert-related critical configuration issue, within a cycle budget. If the budget is exhausted, it builds a partial, low-confidence result instead of failing.

8. Routing​

If topology_validation found a critical alert-related configuration issue, the graph skips runtime investigation and goes straight to analysis (config-remediation path). Otherwise it runs the runtime investigation (runtime path).

9. Final Analysis (analysis)​

Synthesizes the final result, injecting relevant cluster-level and organization-level memory facts and any user-provided context. Recommendations and kubectl commands are tailored to the actual resources and namespaces. The model strategy is per-tenant configurable (AWS Bedrock by default, Amazon Nova for fast extraction/routing, or the direct Anthropic API).

10. Result Delivery​

Results are stored in the database, a notification is sent to Slack, and the investigation is viewable in the portal.

Next Steps​