Skip to main content

Grafana Integration

Overview​

OpsWorker has two distinct Grafana integrations that serve different purposes and are configured independently:

IntegrationDirectionPurposeWhat It Does
Grafana AlertingGrafana → OpsWorkerAlert ingestionSends Grafana alerts to OpsWorker via webhook for automatic investigation
Grafana MCPOpsWorker → GrafanaObservability queriesAllows OpsWorker's AI agents to query Grafana metrics, logs, dashboards, and alerts during investigations and chat

You can use one or both. They are independent: Grafana Alerting works without MCP, and MCP works without Grafana Alerting.

Grafana Alerting (Webhook)​

Receives alerts from Grafana's unified alerting system. Requires Grafana 9+. Grafana delivers alerts in an AlertManager-compatible format, which OpsWorker parses with the same path as Prometheus AlertManager.

How It Works​

  1. Grafana alert rule fires
  2. Notification policy routes it to the OpsWorker contact point (webhook)
  3. OpsWorker receives the alert, normalizes it, and stores it as a signal
  4. If an alert rule matches, an automatic investigation starts

Setup​

1. Get Your Webhook URL and Credentials​

  1. In the OpsWorker portal, go to Integrations → select your cluster → Grafana Alerting and click Install
  2. Copy the webhook URL and the credentials the portal generates

All clusters POST to the same endpoint: https://alerts.ow-saas-prod-ext.opsworker.ai/alerts (dev: https://alerts-dev.opsworker.ai/alerts). The cluster is identified by the credentials, not the URL.

2. Create a Contact Point in Grafana​

  1. In Grafana, go to Alerting → Contact Points
  2. Click New Contact Point
  3. Configure:
    • Name: OpsWorker
    • Type: Webhook
    • URL: Your OpsWorker webhook URL
    • HTTP Method: POST
    • Authorization: scheme Bearer, with the credentials from the portal. This is required, not optional. Without it the webhook is rejected.
  4. Click Test to send a test notification
  5. Click Save

3. Configure Notification Policies​

  1. Go to Alerting → Notification Policies
  2. Either:
    • Add to the default policy: all Grafana alerts go to OpsWorker
    • Create a specific policy: route certain alerts (by label, folder, or severity) to OpsWorker
  3. Set the contact point to OpsWorker
  4. Save

4. Verify​

Trigger a Grafana alert or use the test button on the contact point. Check the OpsWorker portal under Alerts for incoming signals.

Grafana Cloud​

The same setup works with Grafana Cloud: create a webhook contact point using your OpsWorker webhook URL.

Grafana MCP (Observability Queries)​

Enables the Observability AI Agent to query your Grafana instance during investigations and chat sessions. The agent runs as a sidecar alongside the OpsWorker Kubernetes Agent.

What It Enables​

With Grafana MCP configured, OpsWorker's AI agents primarily use list_datasources to discover configured datasources and query_prometheus to run PromQL against your Prometheus datasource (CPU, memory, network, latency, histograms, percentiles). This is how OpsWorker queries Prometheus: there is no standalone Prometheus MCP.

Beyond those two core tools, the Grafana MCP server can expose additional capabilities. Which ones are available depends on your Grafana MCP server version, so treat the following as optional rather than guaranteed:

CapabilityAvailability
PromQL queries (query_prometheus)Core, always available
Datasource discovery (list_datasources)Core, always available
Loki / LogQL log searchDepending on your Grafana MCP server
Dashboard inspectionDepending on your Grafana MCP server
Alert rule and notification policy inspectionDepending on your Grafana MCP server
Incident and on-call browsingDepending on your Grafana MCP server
Annotation retrievalDepending on your Grafana MCP server
Deep link generationDepending on your Grafana MCP server

How It Works​

graph LR
    AI[OpsWorker AI Agent] -->|SQS command| Agent[K8s Agent]
    Agent -->|MCP protocol| MCP[Grafana MCP Sidecar]
    MCP -->|API calls| Grafana[Your Grafana Instance]
  1. The Grafana MCP server runs as a sidecar container in the OpsWorker agent pod
  2. It connects to your Grafana instance using a service account token
  3. During investigations, AI agents send queries via SQS → Agent → MCP → Grafana
  4. Results flow back to the AI for analysis and correlation with Kubernetes data

Setup​

1. Create a Grafana Service Account​

  1. In Grafana, go to Administration → Service Accounts
  2. Click Add service account
  3. Set the role to Viewer (read-only access is sufficient)
  4. Create a token for the service account
  5. Copy the token

2. Enable in OpsWorker Portal​

  1. In the OpsWorker portal, go to Integrations → select your cluster
  2. Select Grafana MCP
  3. Enter your Grafana URL (e.g., https://grafana.example.com)
  4. Save

The integration is stored per cluster with the config {grafana_url, mcp_transport: "sqs_bridge", target: "grafana"}. Queries from the AI reach Grafana over the SQS bridge through the in-cluster agent.

3. Install or Upgrade the Agent with Grafana MCP​

The portal generates a Helm command with the Grafana MCP flags. The agent ships as an OCI chart (no helm repo add needed), the cluster token and API key are auto-filled by the portal, and the Grafana MCP value names are grafana-mcp.grafana.url and grafana-mcp.grafana.apiKey:

helm upgrade --install opsworker-agent oci://public.ecr.aws/k1v3l8a8/opsworker-agent \
  --version=v0.9.3 \
  --namespace opsworker-agent --create-namespace \
  --set clusterToken=<auto-filled-by-portal> \
  --set apiKey=<auto-filled-by-portal> \
  --set grafana-mcp.enabled=true \
  --set grafana-mcp.grafana.url=https://grafana.example.com \
  --set grafana-mcp.grafana.apiKey=YOUR_SERVICE_ACCOUNT_TOKEN \
  --timeout=90s

4. Verify​

After the agent restarts, the Grafana MCP sidecar starts alongside the main agent. Test by asking a question in AI Chat:

What Grafana dashboards exist for the production namespace?

Investigation Enhancement​

When Grafana MCP is active, these investigation agents gain additional capabilities. PromQL via query_prometheus is always available; Loki, incidents, and on-call context depend on your Grafana MCP server version:

AgentGrafana Enhancement
investigateCorrelates alerts with historical metric trends via PromQL; depending on your Grafana MCP server, can also search logs via Loki and inspect related Grafana alerts, incidents, and on-call context
analyze_logsWhere Loki is exposed by your Grafana MCP server, queries Loki for log patterns and cross-references log error spikes with metric anomalies
validate_resourcesChecks CPU/memory utilization metrics from Prometheus to validate resource configurations
check_dependenciesQueries service-level metrics and request latency to identify degrading dependencies

Failure Isolation​

Grafana MCP and the Kubernetes Agent run as independent MCP sessions. If Grafana MCP is unavailable (e.g., Grafana is down), Kubernetes investigation tools continue working normally.

Compatibility​

Grafana VersionAlerting (Webhook)MCP (Queries)
Grafana 9+SupportedSupported
Grafana CloudSupportedSupported
Grafana 8 and earlierNot supported (use legacy alerting → AlertManager → OpsWorker)Supported
Self-hosted GrafanaSupportedSupported (must be reachable from the cluster)

Next Steps​