Self-Healing Kubernetes with AI Agents: From Restart Loops to Real Fixes

What self-healing Kubernetes already does for you

Credit where it’s due. Per the official Kubernetes self-healing docs, the cluster “automatically replaces failed containers, reschedules workloads when nodes become unavailable, and ensures that the desired state of the system is maintained.” That’s a lot of 3 AM pages that never happen.

Here’s the full toolkit, and the fine print on each one:

Most of this only works if you configure it. The probes documentation is clear that liveness probes “determine when to restart a container” and that a startup probe holds off liveness and readiness checks until the app is up. A sane baseline for a stateless service looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: shop
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.42.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-pdb
  namespace: shop
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: checkout

Keep /healthz about the process itself. If your liveness endpoint pings the database, a slow database restarts every healthy Pod at once, and your “self-healing” becomes a self-inflicted outage. Lowkey the most common foot-gun in this whole space.

Where Kubernetes self-healing stops: restarts are not fixes

Hot take: Kubernetes is an incredible reconciler and a terrible debugger. It compares actual state to desired state and closes the gap. It never asks whether the desired state is correct.

Ship an image with a bad config value and the Deployment rolls it out with total confidence. Set the memory limit too low and the container gets OOMKilled, restarted, OOMKilled again. A dependency goes down and your Pods crash on boot until the kubelet backs off into CrashLoopBackOff. The cluster is “healing” the whole time. Nothing is getting fixed.

So stop alerting on single restarts (Kubernetes handled it) and alert on healing loops instead. With kube-state-metrics installed:

# Containers restarting repeatedly: healing without fixing
sum by (namespace, pod, container) (
  increase(kube_pod_container_status_restarts_total[1h])
) > 3

# Containers stuck in CrashLoopBackOff right now
sum by (namespace, pod, container) (
  kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}
) > 0

# Containers whose last termination was an OOM kill
sum by (namespace, pod, container) (
  kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
) > 0

When one of those fires, the human playbook is always the same five commands, run while half-asleep:

kubectl get pods -n shop -o wide
kubectl describe pod <pod-name> -n shop
kubectl logs <pod-name> -n shop --previous
kubectl get events -n shop --sort-by=.lastTimestamp
kubectl rollout history deployment/checkout -n shop

Read the exit reason, read the last crash logs, line it up with the last deploy, decide. That loop of gather, correlate, decide is exactly the part Kubernetes doesn’t do, and exactly the part an AI agent can.

How AI agents make Kubernetes self-healing actually fix things

An AI agent for Kubernetes adds a second control loop on top of the reconciler. Instead of “make actual match desired,” it’s “figure out why this keeps breaking, then change the right thing.” Done properly, it looks like this:

1. 1Detect. A healing-loop alert or SLO burn fires, or a forecast flags a resource heading toward exhaustion.
2. 2Investigate. Agents pull metrics, crash logs, events and recent rollouts in parallel, the same evidence you’d gather by hand.
3. 3Diagnose. A correlation step lines the signals up on one timeline and ranks root-cause hypotheses by evidence.
4. 4Propose. The agent recommends a specific remediation, with its risk and the evidence behind it.
5. 5Gate. Policy decides: run it, ask a human, or hand it over. Blast radius is checked first.
6. 6Verify. After the action, the agent confirms recovery against the original symptom, or rolls back.

This is how HealR’s Agentic War Room for Kubernetes incident investigation is built: a Metrics Agent and a Logs Agent gather signals, a correlation step reconstructs the timeline and assesses impact, an RCA step ranks hypotheses with confidence scores, and a Recommendations step generates remediation steps with a risk assessment. Every conclusion ships with the data behind it, so you can check the agent’s homework.

The vibe check for any “self-healing AI” tool: can it tell you why it wants to roll back, before it rolls back? If not, it’s not an SRE agent, it’s a slot machine with cluster-admin. For the investigation half in depth, read our guide on automating root cause analysis for production incidents.

Auto remediation with approval gates and rollback

Here’s where most teams get nervous, and they should. Google’s SRE Workbook chapter on eliminating toil lays out the safeguards for any automation that touches production: validate inputs, assess the safety of every action before running it (including anything that reduces capacity or redundancy), and default to a human operator when the automation hits an unsafe condition. AI agents don’t get a pass on that. If anything, they need it more.

The practical answer is to gate by risk, not by vibes:

Rollback is the workhorse fix for “we just shipped something bad,” and it comes with its own verification step. Note that Kubernetes won’t roll a bad Deployment back for you; somebody has to decide:

# Roll back to the previous revision, then wait for it to settle
kubectl rollout undo deployment/checkout -n shop
kubectl rollout status deployment/checkout -n shop --timeout=5m

Two more guardrails worth calling out. First, PDBs are not a safety net for agents: the disruptions docs note that deleting Pods or Deployments directly bypasses Pod Disruption Budgets. Second, a fix to one service can hurt the five that depend on it, so check who’s downstream before you act. HealR’s blast radius analysis calculates the impact of a failure across your infrastructure, and TopologyGPT’s dependency mapping keeps that map current instead of a whiteboard photo from last year.

A step-by-step rollout plan for AI agents in Kubernetes

Crawl, walk, run. Nobody should go from zero to “the agent rolls back prod at 3 AM” in one sprint. No cap, the teams that get the most out of AI remediation are the ones that spent a month letting it watch first.

1. 1Fix the basics first: startup, liveness and readiness probes, resource requests and limits, at least two replicas, and a PodDisruptionBudget on every critical service.
2. 2Alert on healing loops, not single restarts: repeated restarts, CrashLoopBackOff and OOMKilled across a window.
3. 3Give the AI agent a read-only ClusterRole (get, list, watch) and keep Secrets out of it.
4. 4Let the agent investigate every page and post evidence plus a ranked root-cause hypothesis to the incident channel.
5. 5Have the agent propose a specific remediation with a risk rating, and require a human to approve it.
6. 6Check the blast radius of every proposed change against your service dependency map before it runs.
7. 7Prefer reversible actions, and define the rollback and the recovery check for each one up front.
8. 8Log every proposal, approval, action and verification result so you can audit it later.
9. 9Graduate individual, low-risk actions to auto-run only after a clean track record your team agrees on.
10. 10Review agent actions in every postmortem and turn repeat fixes into permanent changes.

Step 3 in YAML. Read-only means read-only, and Secrets stay out:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-readonly
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events", "services", "nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["get", "list", "watch"]

The best self-healing incident is the one that never pages. HealR’s Predictive Intelligence forecasting flags resources trending toward exhaustion, which turns some future OOMKills into a daytime ticket to raise a limit. If you want agents tailored to your own runbooks and approval flow, that’s what our custom AI SRE agent solutions are for: human-in-the-loop by default, autonomy widened as trust builds.

How to measure whether self-healing with AI is working

“Autonomous” is not a metric. Toil is. The SRE Workbook defines toil as “the repetitive, predictable, constant stream of tasks related to maintaining a service,” and Google caps the time its SRE teams spend on operational work at 50%. Every 4 AM kubectl logs --previous session is toil with extra steps.

Track these before and after you add agents:

• →Healing loops: count of restart-loop, CrashLoopBackOff and OOMKilled alerts per week.
• →Time to diagnosis: alert fired to first evidence-backed root-cause hypothesis.
• →MTTR: alert fired to verified recovery.
• →Approval rate: share of agent proposals humans approved without edits. This is your trust score for widening autonomy.
• →Repeat incidents: same root cause twice in 30 days means the fix was a restart, not a repair.

MTTR is where leadership will look first. Our breakdown of how to reduce MTTR with AI shows which incident phases eat the minutes, and the ROI calculator turns those minutes into a number your CFO will read.

FAQ: self-healing Kubernetes and AI agents