Suggested Fixes
Overview
Every OpsWorker investigation includes actionable recommendations organized into two categories: immediate actions to resolve the current issue, and preventive measures to stop it from recurring.
Recommendation Categories
Immediate Actions
Steps to fix the problem right now:
- Specific to the current issue and environment
- Include resource names, namespaces, and values from your cluster
- Designed to restore service as quickly as possible
- Typically include kubectl commands you can copy and execute
Example:
Immediate Action: Restart the deployment to recover from the OOM condition:
kubectl rollout restart deployment/api-gateway -n productionThen increase the memory limit to prevent immediate recurrence:
kubectl set resources deployment/api-gateway -n production --limits=memory=512Mi
Preventive Measures
Longer-term fixes to address the root cause:
- Address the underlying issue, not just the symptom
- May involve code changes, configuration updates, or architectural adjustments
- Help break the cycle of recurring alerts
Example:
Preventive Measure: Investigate the connection pool leak identified in the logs. The
maxConnectionssetting in the application config should be bounded. Consider adding a connection pool metrics exporter to catch this trend earlier.
Fixing Wiring Issues
Some of the most actionable fixes target topology wiring issues: mismatches on the connection between two resources, such as a service selector that no longer matches its pods, or a service port that does not line up with the container's target port. OpsWorker flags these during topology and configuration validation as a first-class root-cause category, and the suggested fix points directly at the misconfigured field (for example, "update the app label on the Deployment to match the Service selector").
Contextual Recommendations
Recommendations are tailored to your specific environment:
- Uses actual resource names, namespaces, and configuration values
- Accounts for your cluster's topology (for example, related services that may be affected) and any wiring issues found between resources
- Considers the specific alert type and observed failure pattern
- References real data from the investigation (log lines, event details)
- Is influenced by cluster-level and organization-level memory (what OpsWorker has learned about your environment) and any user-provided context from a manual, free-text investigation
Viewing Recommendations
Recommendations appear in:
- Slack notification: summary with key actions and commands
- Portal investigation detail: full recommendations with context
- Investigation chat: ask follow-up questions about any recommendation
Next Steps
- Command Generation: how kubectl commands are generated
- Safe Execution Model: how OpsWorker ensures safety
- PR Creation: code-level fixes opened as PRs/MRs