The Platform Add-on Layer: What Every Cluster Needs Before Apps Arrive

In this module, you'll learn to:

• List the add-ons a new EKS cluster needs before apps arrive, and what each does
• Give every add-on its own narrowly scoped IAM role
• Configure ExternalDNS and storage classes safely, and avoid two install-script gotchas

Before you start: this builds on Kubernetes on Amazon EKS. It installs the pieces that later modules rely on: load balancing (see Ingress Architecture & Cost), metrics and autoscaling (Scaling & Cost), and logs (Observability). It's based on the add-on setup used in the multi-environment deployment this track draws on, with every name, account, and domain replaced by a placeholder.

A new cluster is an empty building

Terraform can create an EKS cluster and its node groups, but a fresh cluster can't do much yet. It can't create a load balancer for an Ingress, publish DNS names, report pod CPU to an autoscaler, add nodes when it runs out of room, ship logs anywhere, or give pods persistent disks. Each of those is a separate add-on.

It helps to treat these as their own layer: installed right after the cluster exists, the same way in every environment, before any application team deploys anything.

The add-on set in the original deployment

Each add-on gets its own IAM role

Most of these add-ons call AWS APIs, so each needs AWS permissions. The original setup gave every add-on its own IAM role, attached to its own service account in kube-system using IRSA (see Workload Identity & Secrets): one role for the load balancer controller, one for ExternalDNS, one for the autoscaler, one for Fluent Bit. An application that talked to a managed message broker got its own role in the same way.

# One service account per add-on, each pointing at its own role
kubectl create serviceaccount external-dns -n kube-system
kubectl annotate serviceaccount external-dns -n kube-system \
  eks.amazonaws.com/role-arn=arn:aws:iam::<your-account-number>:role/<external-dns-role> \
  --overwrite

# A running pod keeps its old credentials, so restart it to pick up the change
kubectl -n kube-system rollout restart deployment/external-dns

That separation matters. The autoscaler can resize node groups but can't touch DNS; ExternalDNS can edit one hosted zone but can't create load balancers. If one add-on is compromised or misbehaves, the damage stays small.

ExternalDNS across accounts, for a private zone

In a multi-account setup the DNS zone often lives in a shared-services account, not in the cluster's account (the same pattern as the shared Route 53 case study). ExternalDNS can manage records there by assuming a role in the other account:

args:
  - --provider=aws
  - --source=service
  - --source=ingress
  - --domain-filter=internal.example        # only touch records under this domain
  - --aws-zone-type=private                 # only private hosted zones
  - --aws-assume-role=arn:aws:iam::<shared-services-account-number>:role/<dns-role>
  - --policy=upsert-only                    # create and update, never delete
  - --registry=txt
  - --txt-owner-id=<cluster-name>           # marks which records this cluster owns

A few of these flags are there purely for safety:

• --domain-filter and --aws-zone-type=private stop it from ever touching public records or other domains in the same account.
• --policy=upsert-only means it never deletes a record. That's a sensible start, but stale records then need cleaning up by hand.
• --txt-owner-id should be unique per cluster. If several clusters write to the same zone with the same owner ID, they can overwrite each other's records.

Storage classes that respect Availability Zones

StorageClasses, claims, and volumes are introduced in Storage & Other Workload Types. An EBS volume lives in one Availability Zone and can only attach to a node in that zone. If a volume is created before the pod is scheduled, it can end up in a zone where the pod can't run. WaitForFirstConsumer waits until the pod has a node, then creates the volume in that node's zone:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

EFS doesn't have this limit (it's reachable from every zone in the VPC), which makes it a good fit for data that several pods, in different zones, need to share.

Two gotchas from the original install script

1. safe-to-evict goes on the pod, not the Deployment. The script annotated the Cluster Autoscaler Deployment with cluster-autoscaler.kubernetes.io/safe-to-evict: "false", to stop the autoscaler from removing the node it runs on. The command succeeds, but it does nothing: the autoscaler only reads that annotation from pods. It has to go in the pod template:

spec:
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

2. "latest" isn't a version. The script installed some add-ons straight from .../releases/latest/... and .../master/... URLs. Run the same script on two different days and you can get two different versions, so staging and production quietly drift apart. Pin every add-on to a specific version, and change versions on purpose, one environment at a time.

Key terms

• Add-on: cluster software that adds a capability, such as load balancing, DNS, or logs.
• ExternalDNS: creates DNS records for Services and Ingresses.
• metrics-server: reports pod CPU and memory, used by kubectl top and the HPA.
• CSI driver: connects Kubernetes volumes to a storage system, such as EBS or EFS.
• WaitForFirstConsumer: creates a volume only once its pod has a node, in that node's zone.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.