Investigation Lifecycle

Overview

Every investigation moves through the nodes of OpsWorker's adaptive AI graph, from alert arrival to result delivery. There is no standalone "collecting" stage: data gathering happens inside the topology and runtime nodes as they run. Understanding the lifecycle helps you interpret investigation status and troubleshoot issues.

Stages

stateDiagram-v2
    [*] --> Pending: Alert matches rule
    Pending --> InProgress: Investigation starts
    InProgress --> StaticExtraction: static_extraction
    StaticExtraction --> TopologyValidation: topology_validation
    TopologyValidation --> DependencyExtraction: dependency_extraction
    DependencyExtraction --> Investigation: runtime path (no critical config issue)
    DependencyExtraction --> Analysis: config-remediation path (critical config issue)
    Investigation --> Analysis: analysis
    Analysis --> Completed: Results ready
    Analysis --> Partial: Low-confidence synthesis (budget exhausted)
    InProgress --> Failed: Error occurred
    Completed --> [*]
    Partial --> [*]
    Failed --> [*]

Pending

The alert has been received and matched an alert rule. The investigation is queued for processing.

• Duration: seconds
• Visible in: portal investigation list (status: Pending)

In Progress

The investigation is actively running through the graph nodes:

static_extraction

Identifies the affected namespace, pod, severity, and other key fields from the alert payload. Uses fast regex-based extraction first (no LLM call), with AI-based fallback for non-standard or low-confidence formats.

topology_validation

A single LLM node that maps the dependency chain (pod, service, ingress, and upstream owners) via a breadth-first crawl within a Kubernetes call budget AND validates configuration at the same time. It produces resources plus edges, flags wiring issues such as selector or port mismatches and marks whether each is alert-related, and sets has_critical_issues.

dependency_extraction

Builds a service-dependency graph for the affected workloads. Uses Pixie-observed traffic when Pixie is enabled, otherwise falls back to LLM inference.

investigation (conditional)

Runtime analysis of logs, events, and live state, within a cycle budget. This node runs only on the runtime path. If a critical alert-related configuration issue was already found, the graph skips this node and routes straight to analysis.

analysis

Synthesizes the final result: validates and classifies the issue (configuration vs. runtime), identifies the root cause with a confidence level, and generates remediation recommendations. It injects relevant cluster-level and organization-level memory facts and any user-provided context.

Completed

The investigation has finished. Results include:

• Root cause analysis with confidence level
• Affected resources and their topology
• Immediate action recommendations with kubectl commands
• Preventive measure recommendations
• Complete conversation log (the AI's reasoning process)

Notifications are sent to configured Slack channels and the investigation is available in the portal.

Partial Result

If the runtime investigation's cycle budget is exhausted before it reaches a confident conclusion, the analysis node still produces a result, but a low-confidence, partial one. The investigation completes and is delivered with whatever evidence and reasoning was gathered, clearly marked as lower confidence. This is an outcome, not a failure: you still get the topology, the data collected so far, and a best-effort root cause.

Failed

The investigation could not complete. When a node fails, OpsWorker records failed_at_phase so you can see which node it stopped on. Common reasons:

Reason	Resolution
Cluster agent disconnected	Check agent pod status and connectivity
Timeout	Agent may be overloaded, check resource limits
Insufficient permissions	Agent RBAC may not cover the affected namespace

Failed investigations are visible in the portal with error details, including failed_at_phase.

Viewing Investigation Status

• Portal: navigate to Investigations, where each investigation shows its current status
• Slack: Notifications are sent only when investigations complete
• Investigation detail page: Shows the full timeline of each stage

Data Retention

Completed investigations and their collected data are retained in the portal for historical review and trend analysis.

Next Steps

• Investigation Chat: ask follow-up questions
• How Investigations Work: technical deep dive