Metrics & Grafana Capability
Overview
OpsWorker does not have a standalone "observability agent". Instead, Grafana access is delivered as the metrics capability: a set of Grafana MCP tools that OpsWorker agents can call to query metrics, logs, dashboards, alert rules, and incident data. The metrics capability is consumed mainly by the Resource Optimizer chat agent (for right-sizing decisions) and can enrich investigations and other chat agents when present.
These tools are powered by a Grafana MCP sidecar that runs alongside the OpsWorker Kubernetes Agent and communicates with your Grafana instance using a service account token.
Requirements
- Grafana MCP integration configured for the cluster (separate from Grafana Alerting)
- Grafana service account with Viewer role
- See Grafana MCP Setup
When the metrics capability is missing, agents that depend on it (such as Resource Optimizer) are shown as unavailable with their missing capabilities listed.
What the Grafana MCP Tools Provide
| Capability | Description | Examples |
|---|---|---|
| PromQL queries | Execute Prometheus queries via Grafana's Prometheus datasource | CPU usage, memory trends, request rates, latency percentiles, histograms |
| Loki log search | Run LogQL queries via Grafana's Loki datasource | Error pattern detection, log volume analysis, structured log filtering |
| Dashboard inspection | Search, browse, and retrieve data from Grafana dashboards | Find relevant dashboards by service name, view panel data |
| Datasource queries | Query any configured Grafana datasource | Prometheus, Loki, CloudWatch, ClickHouse, Elasticsearch |
| Alert rule inspection | View Grafana alert rules and notification policies | Check which alerts are configured, their thresholds and states |
| Incident browsing | View Grafana incidents and on-call schedules | Check active incidents, who's on call |
| Annotation retrieval | Read Grafana annotations for event correlation | Deployment markers, incident start/end times |
| Deep link generation | Create direct links to Grafana views | Link to specific dashboards, time ranges, panels |
Who Uses the Metrics Capability
- Resource Optimizer (chat agent, requires
cluster+metrics): the primary consumer. It uses actual CPU and memory utilization from Prometheus to right-size requests and limits before proposing a PR/MR. See the AI Agents overview. - Investigations: when the
metricscapability is present, the analysis can be enriched with historical metric trends and log patterns for stronger correlation. - Free-form AI Chat: chat agents can run PromQL/LogQL queries and inspect dashboards on request.
Note: investigate, analyze_logs, validate_resources, and check_dependencies are free-form chat agents, not investigation graph nodes. The investigation graph uses nodes such as topology_validation and analysis (see How Investigations Work).
Use Cases
Enriching Investigations and Chat
- "Prometheus shows memory usage ramping linearly from 12:00 UTC, which correlates with the OOMKill event at 14:23 UTC"
- "Loki logs show 47 connection timeout errors to redis-cache in the 30 minutes before the alert"
- "Grafana alert rule
HighCPUUsagehas been firing intermittently for the past 3 days, a recurring pattern"
In AI Chat
Query your Grafana instance through natural language:
Run a PromQL query: rate(http_requests_total{namespace="production",code=~"5.."}[5m])
Search Loki logs for "connection refused" errors in namespace production
What Grafana dashboards exist for the payment service?
Are there any Grafana alert rules currently firing?
Grafana MCP vs Grafana Alerting
These are separate integrations that serve different purposes:
| Grafana Alerting | Grafana MCP | |
|---|---|---|
| Direction | Grafana to OpsWorker | OpsWorker to Grafana |
| Purpose | Send alerts for investigation | AI queries metrics, logs, dashboards |
| Setup | Webhook contact point in Grafana | Service account + MCP sidecar |
| Requires | Grafana 9+ unified alerting | Any Grafana version with API access |
| Independent | Works without MCP | Works without Alerting |
Using both together provides the richest experience: alerts trigger investigations, and the metrics capability pulls metrics and logs for deeper root cause analysis.
Failure Isolation
The Grafana MCP sidecar and the Kubernetes Agent run as independent MCP sessions. If Grafana is down or the MCP sidecar is unavailable, all Kubernetes investigation tools continue to work normally. The investigation simply has less metrics context.
Next Steps
- Grafana Integration Setup: configure both Grafana integrations
- AI Agents: the full chat agent registry, including Resource Optimizer
- Example Prompts: Grafana-specific prompts to try