OpsWorker: AI-Powered Incident Investigation built on AWS

The Problem We're Solving

When a Kubernetes alert fires at 3 AM, SREs face a familiar drill: SSH into the cluster, grep through logs, trace which services actually depend on each other, and slowly piece together a story of what went wrong - all while half-awake and under pressure to say something useful in the incident channel.

This manual investigation isn't just slow. It's the same work, repeated incident after incident, by whoever happens to be on call - which means the quality of the investigation depends heavily on who got paged. We built OpsWorker to take that first pass off engineers' plates entirely: by the time a human looks at the alert, the investigation is already done.

What OpsWorker Does

OpsWorker is an AI-powered observability platform that automatically investigates Kubernetes alerts. When an alert arrives from Prometheus AlertManager, Datadog, or CloudWatch, OpsWorker springs into action: it extracts resource information, discovers related Kubernetes objects (pods, services, deployments, ingresses), gathers runtime data (logs, events, configurations), and generates a comprehensive root cause analysis with actionable remediation steps. The entire investigation happens in under a minute, 24/7.

Results are delivered instantly via **Slack notifications**, so your on-call engineers get actionable insights right where they work. Engineers can provide feedback on the analysis quality directly in Slack—no context switching required. This feedback loop lets us continuously improve our AI models and investigation strategies—learning from real incidents to get smarter over time.

Human-in-the-Loop with AI Chat

But we didn't stop at automated reports. Our **Chat interface** lets SREs have a conversation with the investigation. Need to dig deeper? Ask follow-up questions like "Why did the pod run out of memory?" or "Show me similar incidents from last week." The chat uses **Claude Haiku** for fast, responsive interactions and can even execute live Kubernetes queries on demand. This human-in-the-loop approach combines the speed of automation with the intuition of experienced engineers.

Built on AWS Serverless & Amazon Bedrock

We're 100% serverless on AWS. Alerts flow through API Gateway → SQS → Lambda for normalization and storage in DynamoDB. EventBridge triggers our investigation workflow, which runs as a Lambda function orchestrating an AI agent graph.

Our Amazon Bedrock model strategy:

• Claude Sonnet - Complex reasoning and root cause analysis in the investigation flow
• Amazon Nova - Fast extraction tasks during alert processing
• Claude Haiku - Responsive, low-latency chat interactions

We communicate with customer Kubernetes clusters via SQS FIFO queues to a lightweight in-cluster agent. The Review UI is served via CloudFront + S3, with a **WebSocket API** for real-time chat streaming.

Why We Chose This Architecture

Serverless was a natural fit—alerts are bursty and unpredictable. We pay only for what we use, scale automatically, and have zero infrastructure to manage. Amazon Bedrock gives us access to best-in-class foundation models without managing ML infrastructure—and lets us pick the right model for each task (Sonnet for depth, Haiku for speed, Nova for extraction). The multi-tenant design (per-org DynamoDB tables, SQS message groups) lets us serve multiple customers securely from a single deployment.