Skip to main content

Prevent Future Incidents

Scenario​

The same type of alert keeps firing. Your team fixes the symptom each time, but the underlying issue persists. You want to break out of the reactive cycle.

How OpsWorker Helps​

Root Cause, Not Just Symptoms​

Every investigation identifies the underlying cause and recommends preventive measures, not just immediate fixes:

  • Immediate: restart the pod to recover.
  • Preventive: fix the memory leak, adjust resource requests, add connection pool bounds.

Acting on the preventive recommendation, rather than only the immediate one, is what stops the alert from coming back.

Daily Digest Deltas​

The daily digest is a per-cluster summary delivered to Slack at 09:00 UTC (also written to S3 and viewable in the portal). It compares the last 24 hours against the prior day, so you get day-over-day (2-day) deltas in alert activity. A namespace whose alert count jumps from one day to the next is a signal worth a closer look.

The digest reports day-over-day movement only. It does not do week-over-week trending or automatic recurring-issue detection.

Proactive Chat Queries​

Use AI Chat to probe the cluster yourself before things escalate:

Which pods have been restarting frequently in namespace production?
Show me deployments with recent OOMKilled events.

Outcome​

  • Break the reactive cycle: preventive recommendations address root causes, not just symptoms.
  • Catch day-over-day spikes: the digest highlights namespaces with rising alert activity.
  • Investigate proactively: ask AI Chat about restart-prone or unhealthy workloads before an alert fires.
  • Fewer repeat alerts: each permanent fix means one less recurring incident.