Prometheus AlertManager Integration

Overview

Configure Prometheus AlertManager to send alerts to OpsWorker for automatic investigation.

Prerequisites

• Prometheus AlertManager running in your cluster or externally
• OpsWorker cluster set up with the Kubernetes Agent installed
• Your OpsWorker webhook URL (available in the portal under Integrations → Alert Manager)

Setup

1. Get Your Receiver Config from the Portal

1. In the OpsWorker portal, go to Integrations
2. Select your cluster
3. Find Alert Manager and click Install to generate the receiver YAML, including the authorization credentials

All clusters POST to the same endpoint: https://alerts.ow-saas-prod-ext.opsworker.ai/alerts (dev: https://alerts-dev.opsworker.ai/alerts). The cluster token is not part of the URL. Instead, the cluster is identified by the credentials the portal puts in the Authorization header, so copy the generated config rather than hand-writing the URL.

2. Configure AlertManager

Add OpsWorker as a webhook receiver in your alertmanager.yml. The http_config.authorization.credentials value comes from the portal:

receivers:
  - name: opsworker
    webhook_configs:
      - url: 'https://alerts.ow-saas-prod-ext.opsworker.ai/alerts'
        send_resolved: true
        http_config:
          authorization:
            credentials: <credentials-from-portal>

route:
  receiver: default
  routes:
    - receiver: opsworker
      match:
        severity: critical
      continue: true  # Also send to other receivers

3. Apply Configuration

If using Prometheus Operator (kube-prometheus-stack), store the credentials in a secret and reference it from authorization:

# AlertmanagerConfig resource
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: opsworker
  namespace: monitoring
spec:
  receivers:
    - name: opsworker
      webhookConfigs:
        - url: 'https://alerts.ow-saas-prod-ext.opsworker.ai/alerts'
          sendResolved: true
          httpConfig:
            authorization:
              credentials:
                name: opsworker-webhook
                key: credentials
  route:
    receiver: opsworker
    matchers:
      - name: severity
        value: critical

For standalone AlertManager, reload the configuration:

# Send SIGHUP to AlertManager
kubectl exec -n monitoring alertmanager-0 -- kill -HUP 1

4. Verify

Trigger a test alert or wait for a real one. Check the OpsWorker portal under Alerts to verify signals are arriving.

Supported Fields

OpsWorker automatically extracts from Prometheus alerts:

Field	Source
Alert name	alertname label
Severity	severity label
Namespace	namespace label
Pod	pod label
Description	description annotation
Summary	summary annotation
All labels	Preserved for filtering

Routing Tips

• Use continue: true to send alerts to both OpsWorker and your existing receivers (PagerDuty, Slack, etc.)
• Route specific severity levels or namespaces to OpsWorker using AlertManager's routing tree
• OpsWorker handles alert grouping independently, so you don't need to adjust AlertManager's group_by settings

Next Steps

• Configure Alert Rules: set up rules for which alerts trigger investigations
• Verify Cluster: confirm end-to-end connectivity