A bad Kubernetes deployment used to mean a scramble: SSH into a node, dig through logs, maybe redeploy from an old Docker tag you hope still exists in your registry. That workflow is mostly gone. Between kubectl rollout undo, Helm’s built-in revision history, and Amazon EKS’s new Version Rollback feature (generally available since July 2026), reverting a broken release is now a scripted, testable operation rather than an emergency ritual. This tutorial walks through building that operation end to end, from a local Minikube sandbox to a production-grade EKS Auto Mode cluster running GPU node pools.

By the end you will have a working rollback playbook covering plain Kubernetes Deployments, StatefulSets, Helm releases, GitOps-managed clusters, and the AWS-specific cluster-level rollback that reverts the control plane itself, not just a workload. We will also cover the sharp edges: database migrations that do not roll back cleanly, StatefulSet ordinal quirks, and the seven-day window AWS gives you before an EKS Version Rollback stops being an option.

Why Kubernetes deployment rollback matters more in 2026

Kubernetes adoption inside container-using organizations sits at 82% in production, according to CNCF survey data, and a growing share of that workload mix is AI training and inference pipelines that are expensive to restart from scratch. When a bad rollout hits a cluster running GPU-backed inference pods, the cost of a slow recovery is not just downtime, it is wasted accelerator-hours. That is the backdrop for why rollback tooling has become a first-class feature rather than an afterthought.

Kubernetes 1.30 shipped Horizontal Pod Autoscaling based on ContainerResource metrics as stable, letting you scale on a single container’s resource usage inside a multi-container pod instead of the whole pod’s aggregate. It also introduced the ImageMaximumGCAge feature gate so kubelet can garbage-collect container images past a maximum age, independent of disk pressure. Both matter for rollback work: a rollback that reverts your deployment but leaves stale HPA behavior or unexpectedly garbage-collected images will not actually restore the state you think it restores. Kubernetes deployment rollback in 2026 is a full-stack concern, not a single command.

On the AWS side, Amazon EKS Version Rollback turns a Kubernetes control-plane upgrade from what AWS’s own announcement calls effectively a one-way door into a reversible operation. Before mid-2026, rolling back a botched EKS control-plane upgrade meant opening a support case or rebuilding the cluster. Now it is an API call, provided you catch the problem within seven days.

Prerequisites and versions used in this guide

Everything below was tested against the following toolchain. Match these versions (or newer within the same minor line) to avoid surprises with flag names and output formatting.

  • kubectl client version 1.30 or newer (run kubectl version --client to check)
  • A Kubernetes cluster on version 1.29-1.33 — either Minikube 1.33+, kind 0.24+, or a managed cluster (EKS, AKS, or GKE)
  • Helm 3.15 or newer if you plan to test Helm-based rollback
  • AWS CLI v2 (2.17 or newer) configured with an IAM identity that has eks:UpdateClusterVersion and rollback permissions, only needed for the EKS-specific sections
  • A container registry you control (Docker Hub, Amazon ECR, or GitHub Container Registry) with at least two tagged image versions to roll between
  • Basic familiarity with YAML manifests and the kubectl apply workflow

You do not need a cloud account to follow steps 1 through 7. The EKS-specific material starts at step 8 and requires an active AWS account with billing enabled, since EKS control planes are not free.

Step 1: Understand how Kubernetes stores revision history

A Kubernetes Deployment does not overwrite pods in place. Every time you change the pod template (a new image tag, a new environment variable, a new resource limit) the Deployment controller creates a new ReplicaSet and scales it up while scaling the old one down. The old ReplicaSet sticks around at zero replicas, and that is exactly what makes rollback fast: undoing a deployment just means telling the Deployment controller to scale the previous ReplicaSet back up and the current one back down. No image pull, no rebuild, no new scheduling decision beyond node placement.

Check how many revisions your cluster currently retains for a given deployment:

kubectl rollout history deployment/checkout-api -n production

By default a Deployment keeps 10 old ReplicaSets via the revisionHistoryLimit field. If that field is unset or set too low, older revisions get pruned and you lose the ability to roll back past a certain point. Set it explicitly in your manifest rather than relying on the default:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: production
spec:
  revisionHistoryLimit: 15
  replicas: 6
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: registry.example.com/checkout-api:1.42.0
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"

Fifteen revisions gives you roughly two to three weeks of rollback headroom on a team that ships several times a day, without bloating etcd with hundreds of stale ReplicaSet objects.

Step 2: Add change-cause annotations so rollback history is readable

Run kubectl rollout history on a deployment with no annotations and you get a list of revision numbers with no context, which is useless at 2 AM. Tag every apply with a change-cause annotation, either manually or through your CI pipeline:

kubectl annotate deployment/checkout-api \
  kubernetes.io/change-cause="deploy v1.42.0 - fix cart total rounding (PR #2201)" \
  -n production --overwrite

kubectl apply -f checkout-api-deployment.yaml -n production

Now kubectl rollout history deployment/checkout-api -n production returns something a human can act on:

REVISION  CHANGE-CAUSE
3         deploy v1.40.0 - add retry logic to payment client (PR #2188)
4         deploy v1.41.0 - bump node base image to 22-alpine (PR #2195)
5         deploy v1.42.0 - fix cart total rounding (PR #2201)

Most teams wire this into their CI system so the annotation is set automatically from the commit message or PR title, so nobody forgets it under deadline pressure.

Step 3: Perform your first rollback with kubectl rollout undo

With history in place, rolling back the checkout-api deployment to the revision immediately before the current one is a single command:

kubectl rollout undo deployment/checkout-api -n production

Kubernetes flips the Deployment’s pod template back to the previous ReplicaSet’s spec and starts a new rolling update in reverse: new pods from the old template come up, old pods from the bad template are terminated, respecting your maxUnavailable and maxSurge settings the whole time. Watch it happen live:

kubectl rollout status deployment/checkout-api -n production

Expected output while the rollback is in progress:

Waiting for deployment "checkout-api" rollout to finish: 2 out of 6 new replicas have been updated...
Waiting for deployment "checkout-api" rollout to finish: 4 out of 6 new replicas have been updated...
deployment "checkout-api" successfully rolled out

If you need to skip past the immediately-previous revision and jump to a specific known-good one, use the --to-revision flag:

kubectl rollout undo deployment/checkout-api -n production --to-revision=3

This is the move you want when the deployment immediately before the bad one was also flawed, which happens more often than teams like to admit.

Step 4: Roll back StatefulSets without breaking ordinal guarantees

StatefulSet rollback uses the same kubectl rollout undo statefulset/<name> syntax, but the mechanics differ in a way that catches people off guard. Deployments replace pods in whatever order the scheduler picks. StatefulSets roll back pods in reverse ordinal order by default, from the highest-numbered pod down to pod-0. If your StatefulSet backs a database cluster with leader election tied to pod identity, rolling back out of order can trigger unnecessary leader elections.

kubectl rollout history statefulset/postgres-replica -n data
kubectl rollout undo statefulset/postgres-replica -n data --to-revision=2

For anything stateful and leader-election-sensitive, set updateStrategy.rollingUpdate.partition before the rollback so only a subset of ordinals get touched first, verify replication health, then lower the partition value to cover the rest. This turns an all-at-once rollback into a staged one, which is the safer default for anything holding data.

Step 5: Roll back Helm releases, not just raw manifests

If your workloads ship through Helm charts rather than raw kubectl apply, use Helm’s own rollback command instead of touching the underlying Deployment directly. Helm tracks its own release history separately from Kubernetes’ ReplicaSet history, and reverting one without the other leaves the two out of sync.

helm history checkout-api -n production
helm rollback checkout-api 4 -n production --wait --timeout 5m

The --wait flag blocks until Helm confirms the rollback’s pods are ready, and --timeout caps how long it will wait before reporting failure, which matters in CI-triggered rollback scripts where you need a clean exit code within a fixed window. Helm’s official documentation covers the full flag set for helm rollback if you need to also revert CRDs or skip hooks during the revert.

Step 6: Handle rollback in a GitOps-managed cluster

Running Argo CD or Flux changes the rollback story because the cluster’s live state is supposed to match a Git repository, not a manually-run command. If you run kubectl rollout undo directly against a GitOps-managed Deployment, the GitOps controller will notice the drift and revert your revert on its next sync cycle, usually within a minute or two.

The correct rollback path in a GitOps setup is a Git revert, not a cluster-side command:

git log --oneline -- manifests/checkout-api/deployment.yaml
git revert <bad-commit-sha> --no-edit
git push origin main

For a genuinely urgent incident where waiting on a normal Git push and sync cycle is too slow, both Argo CD and Flux support pausing auto-sync so you can apply an emergency kubectl rollout undo without it being immediately overwritten, then land the matching Git revert afterward so the two stay in sync. Skipping the follow-up Git revert is one of the most common causes of a rollback that silently un-does itself hours later.

Step 7: Automate rollback triggers with health-based checks

Manual rollback is fine for a team watching a dashboard, but the fastest recovery comes from tying rollback to an automated health signal. A simple version of this uses a post-deploy job that polls an error-rate metric and calls kubectl rollout undo if it crosses a threshold:

#!/usr/bin/env bash
set -euo pipefail

DEPLOYMENT="checkout-api"
NAMESPACE="production"
ERROR_THRESHOLD=5.0
CHECK_WINDOW=120

sleep "$CHECK_WINDOW"

ERROR_RATE=$(curl -s "http://prometheus.internal:9090/api/v1/query" \
  --data-urlencode "query=rate(http_requests_total{app=\"${DEPLOYMENT}\",status=~\"5..\"}[2m])" \
  | jq -r '.data.result[0].value[1] // "0"')

if awk -v er="$ERROR_RATE" -v th="$ERROR_THRESHOLD" 'BEGIN{exit !(er > th)}'; then
  echo "Error rate ${ERROR_RATE} exceeds threshold ${ERROR_THRESHOLD}, rolling back"
  kubectl rollout undo deployment/"$DEPLOYMENT" -n "$NAMESPACE"
  kubectl rollout status deployment/"$DEPLOYMENT" -n "$NAMESPACE" --timeout=180s
  exit 1
else
  echo "Error rate ${ERROR_RATE} within threshold, deployment stable"
fi

Run this as a Kubernetes Job triggered by your CI pipeline immediately after every deploy. It is intentionally simple, a real production setup would use a dedicated progressive-delivery controller like Argo Rollouts or Flagger for canary analysis, but this script demonstrates the underlying loop: deploy, wait, measure, decide, revert if needed.

Step 8: Roll back an EKS cluster’s Kubernetes control-plane version

Everything above rolls back a workload. Amazon EKS Version Rollback, generally available since July 2026, does something different: it rolls back the Kubernetes control plane itself to the previous minor version, and on Auto Mode clusters, the worker nodes too. This is the feature to reach for when the problem is not your application code but the cluster upgrade itself, an incompatible admission webhook, a deprecated API your controllers still call, or a node driver that broke against the new kubelet version.

Before AWS shipped this, reverting a bad EKS control-plane upgrade meant either living with the breakage until you could patch around it, or rebuilding the cluster from scratch. Per AWS’s announcement post, rollback is available at no additional cost and works within a seven-day window following the upgrade, and it is strictly N-to-N-minus-1: if you upgraded from 1.32 to 1.33, you can drop back to 1.32, not further.

Check your cluster’s rollback readiness before you actually need it:

aws eks describe-cluster-versions \
  --cluster-name production-cluster \
  --region us-east-1

aws eks list-insights \
  --cluster-name production-cluster \
  --region us-east-1

The insights call surfaces automated checks covering API deprecation usage, version skew between control plane and nodes, add-on compatibility, and general cluster health, exactly the categories most likely to blow up an upgrade. Run it before you upgrade, not after something breaks.

To trigger an actual rollback after a bad upgrade:

aws eks update-cluster-version \
  --name production-cluster \
  --version 1.32 \
  --region us-east-1

aws eks describe-update \
  --name production-cluster \
  --update-id <update-id-from-previous-command> \
  --region us-east-1

Full step-by-step console and CLI instructions are in AWS’s EKS cluster rollback documentation.

Step 9: Handle rollback on EKS Auto Mode clusters with GPU node pools

EKS Auto Mode adds a wrinkle worth planning for separately: on Auto Mode clusters, AWS rolls back worker nodes before it rolls back the control plane, in that specific order, to avoid a window where new-version nodes are talking to an old-version control plane. According to AWS’s Auto Mode rollback documentation, this node-first sequencing respects your configured Pod Disruption Budgets the same way a normal upgrade does, and a cancel API lets you halt a rollback mid-flight if timing needs to change.

If your Auto Mode NodePools run GPU instances (G-series or P-series, or AWS Trainium) for AI inference or training, verify your Pod Disruption Budgets are tight enough to prevent all GPU pods draining simultaneously, since GPU pod rescheduling is slower and pricier than CPU pod rescheduling:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: gpu-inference-pdb
  namespace: ml-inference
spec:
  minAvailable: 75%
  selector:
    matchLabels:
      workload: gpu-inference

A minAvailable of 75% caps how many GPU pods can be evicted at once during either an upgrade or a rollback, keeping enough capacity online to absorb traffic while nodes cycle. Also worth noting for cost planning: AWS Auto Mode cut management fees for accelerated instances starting July 1, 2026, 35% lower for G-series GPU instances and 60% lower for P-series and Trainium, which changes the math on running redundant GPU capacity during upgrade windows.

Step 10: Watch for and cancel a rollback mid-flight if needed

Sometimes you trigger a rollback and then realize timing is wrong, maybe a batch job is mid-run and node churn right now would kill it. The EKS Version Rollback cancel API lets you stop a node-level rollback in progress without leaving the cluster in a half-upgraded, half-rolled-back state:

aws eks describe-update \
  --name production-cluster \
  --update-id <update-id> \
  --region us-east-1

# If the rollback is still in progress and needs to be halted:
aws eks cancel-cluster-update \
  --name production-cluster \
  --update-id <update-id> \
  --region us-east-1

Cancelling does not undo work already completed on nodes that finished rolling back, it stops the process from progressing further. Always run describe-update immediately after cancelling to confirm the cluster’s actual state rather than assuming it froze cleanly.

Step 11: Verify HPA and image-pull behavior after any rollback

A rollback that only checks “are the pods running” misses regressions in autoscaling and image handling, both of which changed meaningfully in Kubernetes 1.30. If you rolled a cluster back from 1.30 or later to an older minor version, ContainerResource-based HPA metrics and the Downward API’s dual-stack status.hostIPs field will not exist on the older version, so any HPA manifest referencing them needs its own fallback or manual review.

kubectl get hpa -n production -o wide
kubectl describe hpa checkout-api-hpa -n production

Also check image pull timing after a rollback, since Kubernetes 1.30’s image_pull_duration_seconds kubelet metric is a good early signal that a rollback is causing full re-pulls instead of using cached image layers, a symptom of registry or node-cache mismatches introduced during the version change:

kubectl get --raw /api/v1/nodes/<node-name>/proxy/metrics | grep image_pull_duration_seconds

Step 12: Build a complete rollback-tested project

Put the pieces together into one small, testable project. Create a directory with a versioned Deployment, a PodDisruptionBudget, and a rollback script you can run from CI or by hand.

mkdir -p rollback-demo && cd rollback-demo
cat > deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: default
spec:
  revisionHistoryLimit: 10
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: demo-app
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
EOF

kubectl apply -f deployment.yaml
kubectl annotate deployment/demo-app kubernetes.io/change-cause="initial deploy: nginx 1.27" --overwrite

# Simulate a bad deploy
sed -i 's/nginx:1.27/nginx:does-not-exist/' deployment.yaml
kubectl apply -f deployment.yaml
kubectl annotate deployment/demo-app kubernetes.io/change-cause="bad deploy: broken tag" --overwrite

# Confirm it is broken
kubectl rollout status deployment/demo-app --timeout=30s || echo "Deployment stuck, as expected"

# Roll back
kubectl rollout undo deployment/demo-app
kubectl rollout status deployment/demo-app --timeout=60s

Running this end to end gives you a working sandbox to rehearse rollback muscle memory before you need it against a real production incident. Expected final output:

deployment "demo-app" successfully rolled out

Monitor rollback health with Prometheus and Grafana

None of the commands above tell you whether a rollback actually fixed the problem, only whether the pods came back up. You need metrics running before, during, and after the rollback to confirm the underlying issue is gone rather than just restarted. Most teams already run Prometheus for scraping, so the fastest path is a dashboard panel dedicated to rollback windows rather than a general-purpose one.

Add a ServiceMonitor (if you run the Prometheus Operator) that scrapes your application’s error-rate and latency metrics at a tighter interval than your default, since the first sixty seconds after a rollback is when you most need fast feedback:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-api-rollback-watch
  namespace: production
spec:
  selector:
    matchLabels:
      app: checkout-api
  endpoints:
    - port: metrics
      interval: 10s
      path: /metrics

Pair that with a Grafana panel querying the same error-rate expression your automated rollback script uses, so the number a human sees on screen during an incident matches the number the automation is acting on. A common source of confusion during incidents is a dashboard showing one error rate while the automated rollback trigger fired on a differently-windowed query, leading responders to second-guess a rollback that was actually correct.

rate(http_requests_total{app="checkout-api",status=~"5.."}[2m])
/
rate(http_requests_total{app="checkout-api"}[2m])

Track rollback frequency itself as a metric too. A team rolling back more than once or twice a month usually has a deeper problem in its testing or review pipeline that no amount of rollback tooling fixes, it just makes the symptom less painful.

Use Argo Rollouts for automated canary-based rollback

The manual health-check script in step 7 works, but it is a blunt instrument: it checks the whole deployment after the fact rather than catching a problem while only a fraction of traffic is exposed to it. Argo Rollouts replaces the standard Deployment resource with a Rollout custom resource that supports canary and blue-green strategies with automated analysis steps built in.

Install the controller and swap your Deployment manifest for a Rollout with a canary strategy that ramps traffic in stages, pausing for an AnalysisRun between each step:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
  namespace: production
spec:
  replicas: 6
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 120 }
        - analysis:
            templates:
              - templateName: error-rate-check
        - setWeight: 60
        - pause: { duration: 120 }
        - setWeight: 100
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: registry.example.com/checkout-api:1.42.0

If the AnalysisRun’s error-rate query breaches its threshold at any step, Argo Rollouts automatically aborts and scales the canary back to zero, restoring 100% of traffic to the stable ReplicaSet without a human touching a keyboard. Trigger the same behavior manually if you spot a problem before automated analysis catches it:

kubectl argo rollouts abort checkout-api -n production
kubectl argo rollouts undo checkout-api -n production
kubectl argo rollouts get rollout checkout-api -n production --watch

The practical benefit over plain kubectl rollout undo is exposure: a canary catches a bad release at 20% of traffic instead of 100%, so by the time a rollback fires, far fewer users ever saw the broken version. For workloads processing payments or handling user data, that exposure difference is often the entire point of adopting progressive delivery over a straight rolling update.

Rolling back Ingress and Service networking changes

Workload rollback gets most of the attention, but a meaningful share of production incidents trace back to a networking change: an Ingress annotation update, a Service selector edit, or a new NetworkPolicy that quietly blocks traffic it should not. None of these live inside a Deployment’s revision history, so kubectl rollout undo does nothing for them, and teams sometimes waste the first ten minutes of an incident rolling back the wrong resource entirely.

Ingress, Service, and NetworkPolicy objects do not carry built-in revision history the way Deployments do, so your rollback path for these depends entirely on how they are managed. If they are applied through kubectl apply from version-controlled YAML, keep a local copy of the last-known-good manifest and diff before reapplying:

kubectl get ingress checkout-api -n production -o yaml > current-ingress.yaml
diff current-ingress.yaml last-known-good-ingress.yaml
kubectl apply -f last-known-good-ingress.yaml -n production

If you manage these resources through Helm or a GitOps controller, treat them exactly like the application manifests in steps 5 and 6: a Helm rollback reverts the whole chart including Ingress and Service templates together, and a GitOps setup reverts them the moment a Git revert lands, since they live in the same repository path. The mistake to avoid is manually patching an Ingress annotation by hand during an incident and then forgetting it exists outside version control entirely, which guarantees the next legitimate deploy silently reverts your emergency fix.

For NetworkPolicy specifically, keep a minimal “known good” baseline policy on hand that you can apply immediately if a new policy is suspected of blocking legitimate traffic, buying time to debug the actual rule without leaving production fully locked down or fully open:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: checkout-api-baseline
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: checkout-api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector: {}
      ports:
        - protocol: TCP
          port: 8080

Applying this baseline immediately restores traffic from anywhere in the cluster on port 8080 while you diagnose the more restrictive policy that caused the outage, then you replace it with a properly scoped rule once the real fix is understood.

Common pitfalls when rolling back Kubernetes deployments

  • Assuming database migrations roll back with the app. Kubernetes reverts pods, not schemas. A migration that dropped a column will break the old application code the moment it tries to read that column. Always version migrations separately and confirm backward compatibility before rolling the app back.
  • Fighting a GitOps controller. Running kubectl rollout undo against an Argo CD or Flux-managed resource without pausing sync gets silently reverted on the next reconciliation loop, often within a minute.
  • Letting revisionHistoryLimit default too low. If old ReplicaSets get pruned before you notice a slow-burn regression, there is nothing left to roll back to. Set the limit explicitly.
  • Skipping change-cause annotations. A revision list full of unlabeled numbers is useless during an incident when you need to pick the right target fast.
  • Rolling back StatefulSets without considering ordinal order. Reverse-ordinal rollback can trigger unwanted leader elections in clustered stateful workloads.
  • Forgetting the seven-day EKS rollback window. Amazon EKS Version Rollback only works within seven days of the original upgrade. Past that, you are back to manual remediation.
  • Not testing HPA behavior after a cross-version rollback. Metrics and fields introduced in a newer Kubernetes minor version silently stop working on an older one.
  • Treating a config-only rollback as free. Even a “just flip a flag” rollback still triggers a full rolling restart of pods unless you specifically design for hot-reload, which means the same disruption budget math applies.

Kubernetes rollback methods compared

MethodScopeSpeedBest for
kubectl rollout undo (Deployment)Single workloadSeconds to minutesStateless app rollback
kubectl rollout undo (StatefulSet)Single stateful workloadMinutes (ordinal-ordered)Databases, message queues
helm rollbackFull chart releaseMinutesMulti-resource chart deploys
Git revert (GitOps)Anything Git-managedMinutes (sync-cycle dependent)Argo CD / Flux clusters
EKS Version Rollback (cluster)Control plane + Auto Mode nodesTens of minutesBroken control-plane upgrades

EKS Version Rollback: key facts and constraints

AttributeDetail
AvailabilityGenerally available since July 2026, all EKS regions
Rollback window7 days from the original upgrade
Version scopeN to N-1 minor version only
Auto Mode node orderWorker nodes roll back before the control plane
CostNo additional charge for the rollback operation itself
GPU management fee changeG-series down 35%, P-series/Trainium down 60% (effective July 1, 2026)
CancellationSupported via cancel API while in progress

Troubleshooting rollback issues

  • Rollback command succeeds but old bug reappears immediately. Check if a GitOps controller resynced the bad manifest right after your manual undo. Pause sync before manual rollback, then land a matching Git revert.
  • kubectl rollout undo returns “no rollout history found.” Your revisionHistoryLimit likely pruned the target revision, or the deployment was only ever applied once. Check with kubectl rollout history first.
  • Rollback appears to hang at “Waiting for deployment rollout to finish.” A readiness probe on the reverted image may be failing for an unrelated reason (dependency outage, DNS issue). Run kubectl describe pod on the new pods to see events.
  • StatefulSet rollback stalls on one specific ordinal. Check if a PersistentVolumeClaim tied to that ordinal has a scheduling conflict or is stuck in a previous binding.
  • Helm rollback fails with “another operation is in progress.” A prior Helm operation left a release in a pending state. Run helm history to find the stuck revision, then helm rollback with a specific revision number rather than the implicit previous one.
  • EKS rollback rejected with a version-window error. You are outside the seven-day window. Confirm the original upgrade timestamp with aws eks describe-update.
  • EKS Auto Mode GPU nodes drain slower than expected during rollback. Your PodDisruptionBudget’s minAvailable may be too permissive, allowing large batches to drain at once. Tighten it and re-run.
  • HPA stops scaling after a cross-version rollback. ContainerResource-based metrics from Kubernetes 1.30+ do not exist on older minor versions. Check kubectl describe hpa for a metric-not-found condition and fall back to Resource-based metrics.

Advanced tips for production rollback strategy

Pair rollback tooling with progressive delivery so most bad deploys never reach 100% of traffic in the first place. Argo Rollouts and Flagger both support canary analysis that automatically halts and reverts a rollout the moment error-rate or latency metrics cross a threshold, which turns the manual health-check script from step 7 into a first-class Kubernetes resource with its own controller loop.

For clusters running EKS Auto Mode with EFA-enabled GPU node pools for distributed training, treat rollback rehearsal as part of your change-management process, not just an incident response tool. Schedule a quarterly game-day where you deliberately trigger an EKS Version Rollback against a staging cluster running representative GPU workloads, timing how long node-level rollback actually takes with your specific PodDisruptionBudget settings. The seven-day window is generous, but you do not want the first real rollback attempt to be during an actual outage.

Finally, separate your rollback runbook by blast radius. A single Deployment rollback is low-risk and can reasonably be automated with minimal human approval. A cluster-level EKS Version Rollback touches every workload on the cluster and should require an explicit sign-off step, even when the tooling makes it technically a single command.

Frequently asked questions

Does kubectl rollout undo pull a new container image?
No. It repoints the Deployment at the previous ReplicaSet, which already has its pod template and image reference defined. If that image is still present in the node’s local cache, no pull happens at all. If not, a normal pull runs against whatever registry the tag points to.

Can I roll back more than one revision at a time?
Yes, using kubectl rollout undo deployment/<name> --to-revision=<N> where N is any revision still retained under your revisionHistoryLimit.

What happens to a rollback if revisionHistoryLimit already pruned the target revision?
The command fails with an error indicating no matching revision exists. There is no recovery path once a ReplicaSet object has been garbage-collected. You would need to reapply the old manifest from source control instead.

Is Amazon EKS Version Rollback available on every EKS cluster?
It is generally available across all regions where EKS runs, at no extra charge for the rollback action itself, but it only covers a single minor version step back and only within seven days of the original upgrade.

Does rolling back a Kubernetes Deployment also roll back its ConfigMaps and Secrets?
Not automatically. ConfigMaps and Secrets are separate objects. If your pod template references a ConfigMap by a fixed name rather than a hash-suffixed name generated per revision, rolling back the Deployment will not restore an older ConfigMap version unless you version and reference ConfigMaps explicitly.

Should I roll back or roll forward with a hotfix?
It depends on how fast a fix can be validated. Rollback is near-instant and low-risk if the previous revision was stable. Rolling forward with a hotfix is preferable when the previous revision had its own known issues, but it takes longer to prepare and test.

Can Helm and kubectl rollback histories get out of sync?
Yes, if you mix Helm-managed deploys with direct kubectl apply commands against the same resources. Pick one management path per workload and stick to it to avoid the two histories diverging.

What is the safest way to test a rollback without affecting production?
Reproduce the deployment in a staging namespace or cluster with the same revisionHistoryLimit, replica count, and PodDisruptionBudget settings, then run through the same undo commands before ever touching production. The complete project in step 12 is designed to be that sandbox.

How long should I wait after a rollback before declaring the incident resolved?
There is no universal number, but most teams watch error rate and latency for at least two full traffic cycles after the rollback completes, long enough to cover any cron jobs, cache warm-ups, or connection-pool resets that only surface a few minutes in. A rollback that looks clean at the one-minute mark can still regress once a delayed batch job or a slow-draining connection pool catches up.