Storage & Other Workload Types: Data That Outlives a Pod

In this module, you'll learn to:

• Explain why a pod's files don't survive it, and when that matters
• Give a pod persistent storage with a PersistentVolumeClaim and a StorageClass
• Choose between a Deployment, StatefulSet, DaemonSet, Job, and CronJob

Before you start: this builds on Pods, Deployments & Rollouts, especially the idea that pods are disposable, and on Services & Cluster Networking for headless Services.

A container's files die with the pod

Everything a container writes to its own filesystem is lost when the container restarts or the pod is replaced. For most web services that's exactly right: they keep their data in a database, not on local disk, and that's what makes them easy to replace, move, and scale.

Some workloads do need data to survive: a database you run yourself, a message broker, an app that stores uploaded files. Kubernetes handles that by keeping the storage separate from the pod, so a new pod can pick up the same disk the old one used.

From scratch space to persistent disks

A volume is a directory that's mounted into a container. The volume type decides where the data actually lives and how long it lasts:

Three objects, one disk

Persistent storage is split into three objects, so that the people writing apps don't need to know how disks are created:

• StorageClass: how disks are made. Set up once per cluster by the platform team: which driver, which disk type, which zone rules.
• PersistentVolumeClaim (PVC): a request for storage, written by the app team: "10 GiB, readable and writable by one node".
• PersistentVolume (PV): the actual disk, created automatically by the StorageClass's driver to satisfy the claim.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: orders-data
spec:
  accessModes: [ReadWriteOnce]   # one node at a time can mount it read-write
  storageClassName: standard     # kind's built-in class; gp3 or similar on EKS
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: writer
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "date >> /data/log.txt; sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: orders-data

The access mode matters more than it looks. ReadWriteOnce means one node at a time, which is what block disks such as Amazon EBS support. ReadWriteMany lets pods on many nodes share the volume, which needs a network file system such as Amazon EFS. On AWS, an EBS volume also lives in a single Availability Zone, which is why The Platform Add-on Layer sets its StorageClass to wait until the pod is scheduled before creating the disk.

For apps that need a stable identity

A Deployment treats its pods as identical and interchangeable: random names, any pod can be replaced by any other, and they all share one PVC if they mount one. Clustered software such as databases and brokers can't work like that. Each member needs its own disk and a name that stays the same when it restarts, so the others can find it.

A StatefulSet provides both:

• Pods are named in order, orders-db-0, orders-db-1, orders-db-2, and keep those names when they're replaced.
• Each pod gets its own PVC from a volumeClaimTemplates section, and gets the same PVC back after a restart.
• Pods start one at a time, in order, and stop in reverse order.
• With a headless Service, each pod gets its own DNS name, such as orders-db-0.orders-db.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: orders-db
spec:
  serviceName: orders-db         # a headless Service (clusterIP: None) with this name
  replicas: 3
  selector:
    matchLabels: { app: orders-db }
  template:
    metadata:
      labels: { app: orders-db }
    spec:
      containers:
        - name: db
          image: <database-image>
          volumeMounts:
            - name: data
              mountPath: /var/lib/data
  volumeClaimTemplates:          # one PVC per pod: data-orders-db-0, -1, -2
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 20Gi

Running a production database on Kubernetes is possible, but it makes backups, upgrades, and failover your job. On AWS, most teams use a managed service such as Amazon RDS or Aurora for the database, and keep the cluster for stateless apps. Learn StatefulSets anyway: plenty of add-ons and tools you'll install use them.

Picking the right controller

# A CronJob that runs a report every night at 02:00, in the control plane's
# time zone (usually UTC) unless you set spec.timeZone
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid      # don't start a new run while one is still going
  jobTemplate:
    spec:
      backoffLimit: 2            # retry a failed run twice, then give up
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: report
              image: busybox:1.36
              command: ["sh", "-c", "echo building report; sleep 5"]

Prove the data survives

On the kind cluster from Your First Cluster, which comes with a standard StorageClass:

# Save the PVC and Pod from above as storage.yaml, then:
kubectl apply -f storage.yaml
kubectl get pvc,pv

# Read the file, delete the pod, recreate it, and read it again:
# the first line is still there, and a second one has been added
kubectl exec writer -- cat /data/log.txt
kubectl delete pod writer --now
kubectl apply -f storage.yaml
kubectl exec writer -- cat /data/log.txt

# Run a Job once and read its output
kubectl create job hello-job --image=busybox:1.36 -- echo "job done"
kubectl get jobs
kubectl logs job/hello-job

# Clean up; deleting the claim deletes the disk with this StorageClass
kubectl delete -f storage.yaml
kubectl delete job hello-job

Key terms

• Volume: a directory mounted into a container; its type decides how long the data lasts.
• PersistentVolumeClaim (PVC): a request for storage of a given size and access mode.
• PersistentVolume (PV): the actual disk that satisfies a claim.
• StorageClass: how disks get created: the driver, the disk type, and binding rules.
• StatefulSet: pods with stable names and their own disks.
• DaemonSet: exactly one pod on every node.
• Job and CronJob: pods that run to completion, once or on a schedule.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.