How Investigations Work

Overview

An OpsWorker investigation is an automated, AI-driven analysis of a Kubernetes alert. When an alert arrives, an adaptive graph of AI nodes works together to discover affected resources, gather data, identify the root cause, and generate actionable remediation steps. The flow is branching, not strictly linear: depending on what the early nodes find, the investigation can take a config-remediation path or a runtime-analysis path.

Investigation Flow

flowchart TD
    A[Alert arrives via webhook] --> B[Normalize and evaluate alert rules]
    B --> C[static_extraction: regex field extraction, AI fallback if low confidence]
    C --> D[topology_validation: BFS topology crawl plus config validation in one node]
    D --> E[dependency_extraction: service dependency graph via Pixie, LLM fallback]
    E --> F{Critical alert-related config issue?}
    F -->|Yes| H[analysis: final synthesis]
    F -->|No| G[investigation: conditional runtime analysis]
    G --> H
    H --> I[Deliver to Slack and Portal]

Nodes

1. Alert Normalization and Rule Evaluation

Incoming alerts from Prometheus AlertManager, Grafana Alerting, and Datadog are converted into a common format. OpsWorker recognizes the AlertManager and Datadog webhook payload formats natively; sources like Grafana Alerting that emit AlertManager-compatible payloads work without custom code paths. The normalized signal is evaluated against your alert rules, and a match (with auto-investigation enabled) starts the graph.

2. Field Extraction (static_extraction)

OpsWorker identifies key fields from the alert:

• Namespace: which Kubernetes namespace is affected
• Pod/Resource: the specific resource that triggered the alert
• Severity: alert severity level
• Description: human-readable context

Fast regex-based extraction runs first with no LLM call. If confidence is low (for example, a non-standard alert format), AI-based extraction provides a fallback.

3. Topology and Configuration Validation (topology_validation)

This is a single LLM node that does two things together. Starting from the alerting resource, it performs a breadth-first topology crawl within a bounded Kubernetes call budget, AND it validates configuration as it goes. It produces the set of affected resources plus the edges between them, flags wiring issues (such as selector or port mismatches) and marks whether each issue is alert-related, and sets a has_critical_issues signal.

graph LR
    Deployment --> Pod
    ReplicaSet --> Pod
    Pod --> Service
    Service --> Ingress
    Pod --> ConfigMap
    Pod --> Secret[Secret metadata]

This step is critical: the root cause often is not in the alerting resource itself, but in an upstream or downstream dependency. For example, a pod crash might be caused by a misconfigured service selector or a missing configmap.

4. Dependency Extraction (dependency_extraction)

OpsWorker builds a service-dependency graph for the affected workloads. When Pixie is enabled it uses observed service-to-service traffic; otherwise it falls back to LLM-based inference from the topology and configuration.

5. Runtime Investigation (investigation, conditional)

This node performs runtime analysis (logs, events, live state) and only runs when there is no alert-related critical configuration issue. It works within a cycle budget. If the budget is exhausted before it reaches a confident conclusion, it builds a partial, low-confidence result rather than failing.

Routing: if topology_validation found a critical alert-related configuration issue, the graph skips runtime investigation and goes straight to analysis (the config-remediation path). Otherwise it runs the runtime investigation (the runtime path).

6. Final Analysis (analysis)

The analysis node synthesizes everything into the final result. It injects relevant cluster-level and organization-level MEMORY facts and any user-provided context (for manual, free-text investigations) so recommendations reflect what OpsWorker already knows about your environment. The model strategy used across these nodes is per-tenant configurable (see below).

Data Collected Along the Way

As the nodes run, OpsWorker gathers:

Data Type	Source	Example
Logs	Container stdout/stderr	Application errors, stack traces
Events	Kubernetes events	Pod scheduling, OOM kills, image pulls
Configuration	Resource specs	Deployment config, resource limits, env vars
Status	Resource status	Pod phase, container states, restart counts
Endpoints	Service endpoints	Healthy/unhealthy backends

When additional integrations are configured, the data expands:

Data Type	Integration	Example
Prometheus metrics	Grafana MCP	CPU/memory trends, request rates, latency via PromQL
Application logs	Grafana MCP (Loki)	Log patterns, error rates via LogQL
Code changes	GitHub / GitLab	Recent commits, PRs correlated with incident timeline

Note: Datadog is an alert source only. OpsWorker does not query Datadog during an investigation.

Model Strategy

OpsWorker's model strategy is per-tenant configurable. The default uses AWS Bedrock (default model us.anthropic.claude-3-5-sonnet-20241022-v2:0), with Amazon Nova available for fast extraction and routing, or you can configure the direct Anthropic API. It is not a fixed multi-model ensemble; each tenant chooses how models are assigned to the nodes.

What the Analysis Produces

Based on the synthesis, OpsWorker generates:

• Root cause statement: what went wrong and why
• Immediate actions: steps to fix the current issue, with specific kubectl commands
• Preventive measures: longer-term changes to prevent recurrence

Supported Alert Types

Alert Type	Examples	Investigation Approach
Pod failures	CrashLoopBackOff, ImagePullBackOff, OOMKilled	Logs, events, resource limits, exit codes
Service issues	No endpoints, connection refused	Service selectors, pod readiness, endpoint health
Ingress problems	5xx errors, TLS failures	Ingress config, backend service, certificate status
Resource exhaustion	CPU throttling, memory pressure	Resource limits vs usage, HPA config
Deployment issues	Failed rollout, stuck rollout	Deployment strategy, pod scheduling, image availability

Time to Complete

Most investigations complete in under 2 minutes from alert arrival to Slack notification. Investigation time depends on:

• Number of resources discovered in the topology
• Volume of logs to analyze
• Cluster agent response time

Next Steps

• Automatic Investigations: how auto-investigation works
• Investigation Lifecycle: stages of an investigation
• Root Cause Analysis: how root causes are identified