Jessica Hsiao
Allie Rittman
Ivan Ilichev
As teams adopt Kubernetes at greater scale, they face increased complexity in keeping their growing list of workloads and services up and running. Achieving the visibility and context needed to detect and resolve incidents quickly is difficult amid a constant flood of telemetry data and alerts. Furthermore, Kubernetes expertise often remains siloed in DevOps and infrastructure teams. Application development teams frequently lack the relevant Kubernetes expertise to confidently investigate and mitigate issues on their own, resulting in platform engineers becoming a support bottleneck for issue resolution.
Datadog Bits AI Kubernetes Remediation, now generally available, addresses these problems by helping you identify and fix common infrastructure issues in your Kubernetes clusters before they escalate into business-impacting incidents. It combines AI-powered root cause analysis and recommendations with automated workflows, helping you accelerate investigations and immediately take action on issues to reduce mean time to resolution (MTTR).
In this post, we’ll explain how Bits AI Kubernetes Remediation can help you:
Troubleshoot Kubernetes issues more efficiently
Kubernetes errors such as CrashLoopBackOff, OOMKilled, and ImagePullBackOff can stem from root causes that are several layers deep. Properly diagnosing these errors requires expertise that can take years to acquire. In addition, a Kubernetes-related alert for a large cluster often leads to a time-consuming review of all the related telemetry data, which further complicates troubleshooting efforts. If the errors are widespread among pods, it can indicate that the associated services that are hosted within those pods are failing.
When these common Kubernetes errors occur, Bits AI Kubernetes Remediation detects the issue and generates remediation guidance for each problematic workload. Aggregating all the related troubleshooting information in one place reduces the time spent on gathering context, which enables teams to respond to issues more quickly and get ahead of potential service disruptions.
Context-rich AI feedback also makes it easier to understand why an issue occurred and how to fix it. Suppose that one of your workloads is experiencing a CrashLoopBackOff error that is causing your Datadog monitor to trigger an alert. You can view the AI-powered remediation guidance directly in the alert by clicking on the affected workload, by using the Kubernetes Remediation tab, or by selecting an impacted pod in the Kubernetes Explorer. At a glance, you can quickly understand the situation by reviewing an explanation of what happened to trigger the alert, the recommended next steps to resolve the issue, and any key contextual information. These remediation recommendations are also integrated with Datadog Software Catalog and provide helpful details such as relevant code repositories and teams to page.
In the example in the following screenshot, Bits AI Kubernetes Remediation detects that the specified container command could not be found and has caused a container to terminate, resulting in the pod being stuck in a restart loop. The analysis from Bits AI refers to errors in the application logs and explains that the problem was caused by a misconfigured command in the manifest.
To safely stabilize the issue, the remediation recommendation is to perform a rollback to a stable version of the Kubernetes workload manifest. Alternatively, you can review AI-suggested next steps to resolve the issue. In the following screenshot, Bits AI has identified a fix: Correct the typo in the container command, and apply the changes to the deployment manifest.
Efficiently stabilize your environment with guided actions
In addition to providing analysis and recommended next steps, Bits AI Kubernetes Remediation enables you to take direct action from a recommendation. Consider another example, where you have a Kubernetes workload with pods that are experiencing CrashLoopBackOff errors because the pods are repeatedly reaching memory limits. You can review the workload’s historical memory usage against the existing threshold and update the memory limit to a higher value with confidence.
After you preview the changes, you can immediately apply the fix by creating a GitHub pull request (PR) directly from the recommendation by using the Bits AI Dev Agent (currently in Preview). With the GitHub integration enabled with proper Bits AI Dev Agent permissions, you can select the repository that contains your configuration definitions.
Clicking the Fix with Bits button starts a Bits AI Dev Agent code session, where the agent will automatically identify and update the right configuration file in your desired repository and generate a PR.
With this approach, you can close the loop on a Kubernetes issue in minutes instead of taking hours to manually troubleshoot the problem, review telemetry data, and apply the necessary changes.
Implement faster, smarter issue detection and response today
Bits AI Kubernetes Remediation empowers your organization to respond to incidents and take corrective action more quickly and confidently. AI-powered reasoning and recommendations enable teams of all experience levels to understand underlying problems and accelerate root cause analysis and fixes. As your teams become more efficient at addressing complex Kubernetes issues, reduced MTTR leads to fewer escalations and better uptime for your customers.
Bits AI Kubernetes Remediation is included with Datadog Container Monitoring. To create PRs from recommendations with the Bits AI Dev Agent, sign up for the Kubernetes Remediation Preview. For more information, check out the Bits AI Kubernetes Remediation documentation.
If you don’t already have a Datadog account, you can sign up for a 14-day free trial to get started.
