Cluster Upgrades Without Drama

In this module, you'll learn to:

• Explain why small, regular upgrades are safer than rare, big ones
• Find APIs the new version removes before you upgrade
• Upgrade the control plane, add-ons, and nodes in the right order, rehearsed outside Production

Before you start: you'll want a non-production cluster to rehearse on, and your cluster version defined in code (Terraform or similar).

Upgrades are routine work, so make them boring

Kubernetes ships a new minor version roughly three times a year, and each EKS version gets a fixed window of standard support. After that, a cluster moves to extended support, which costs several times the standard per-cluster price, and eventually it's upgraded for you whether you're ready or not.

The teams that struggle with upgrades are usually the ones that do them rarely. A small, repeatable upgrade every few months is far less risky than a big jump once the deadline arrives.

Two rules shape everything else:

• The control plane moves one minor version at a time. You can't go from 1.28 straight to 1.31. Each step is its own upgrade.
• Nodes can lag behind the control plane, never lead it. Upgrade the control plane first, then bring the nodes up to match.

Find what the new version removes

Most upgrade failures aren't caused by the upgrade itself. They come from manifests or Helm charts that still use an API version the new release has removed. The cluster upgrades fine, then the next deploy fails.

# Check the EKS upgrade insights for the cluster
aws eks list-insights --cluster-name <cluster>

# See which API versions the cluster currently serves
kubectl api-versions

EKS upgrade insights flag deprecated API usage the cluster has actually seen, which is more reliable than grepping a repository. Tools such as pluto can scan your manifests and Helm releases as well. Fix everything they find before you start. Changing an API version on a live cluster is easy; finding out after the upgrade is not.

Control plane, then add-ons, then nodes

Upgrade one minor version at a time (for example 1.32 to 1.33), in this order:

1. Control plane. Run aws eks update-cluster-version, or change the version in Terraform. EKS replaces the API servers in the background, and running workloads keep going.
2. Core add-ons. Move the VPC CNI, CoreDNS, and kube-proxy to versions made for the new release. For managed add-ons this is a version bump; self-managed ones need their manifests updated.
3. Nodes. For managed node groups, start a version update, and EKS replaces the nodes one at a time, cordoning and draining each. With Karpenter, updating the AMI or the node class makes it replace nodes gradually. Until this step finishes, the nodes run one version behind the control plane, which is allowed.
4. Everything else. Controllers that talk to the Kubernetes API, such as the AWS Load Balancer Controller, an ingress controller, or Argo CD, have their own supported version ranges. Check each against the new version.

Then repeat for the next minor version if you're more than one behind.

# Watch nodes come up on the new version as the rollout proceeds
kubectl get nodes -L eks.amazonaws.com/nodegroup \
  -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion

Prove it on a non-production cluster first

Run the same upgrade on Dev or Staging a week or two before Production, with real deploys happening on it in between. Most surprises show up as a failed deploy or a controller that stops reconciling, and you want to meet those somewhere quiet.

Key terms

• Minor version: a Kubernetes release such as 1.32 or 1.33; upgrades go one at a time.
• Standard and extended support: EKS's support window for a version, and the more expensive period after it.
• Upgrade insights: EKS checks that flag deprecated APIs your cluster has actually used.
• Cordon and drain: stop new pods landing on a node, then evict the pods already on it.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.