How a Kubernetes Cluster Actually Works

In this module, you'll learn to:

• Explain the reconciliation loop: desired state, actual state, and controllers
• Name the parts of the control plane and of a worker node, and what each does
• Trace what happens between kubectl apply and a running container

Before you start: finish Part 1, especially Your First Cluster, where you watched Kubernetes replace a deleted pod. This module explains how that happened. Keep the kind cluster running; the commands at the end use it.

You describe what you want, and Kubernetes keeps making it true

With a plain server, you give instructions: start this process, copy that file, restart the service. If the process crashes at 3 a.m., it stays down until something or someone restarts it.

Kubernetes works the other way round. You write down the desired state ("three copies of the orders-api image, version 1.4, each with half a CPU"), and you hand it to the cluster. From then on, small programs called controllers keep comparing what you asked for with what's actually running, and fix any difference they find. A crashed container gets replaced. A lost node's pods get rescheduled somewhere else. You never tell the cluster how to recover; you only told it what should exist.

Almost everything in this track is this same loop applied to a different problem. A Deployment controller keeps the right number of pods running. A load balancer controller keeps a cloud load balancer matching your Ingress. Argo CD keeps the cluster matching Git. Once this idea clicks, the rest of Kubernetes stops looking like magic.

The control plane decides, the nodes do the work

A cluster has two halves.

The control plane is the brain. It stores the desired state, decides where things should run, and runs the controllers. Its main parts:

• API server: the front door. Every tool, including kubectl, the nodes, and every controller, reads and writes cluster state only through it.
• etcd: the database where the API server keeps all cluster state.
• Scheduler: picks a node for each new pod, based on the resources the pod requests and the rules it sets.
• Controller manager: runs the built-in controllers (Deployments, ReplicaSets, Jobs, and others).

The worker nodes are the muscle. They're ordinary machines (a Docker container in a local kind cluster, EC2 instances on Amazon EKS) that actually run your containers. Each node runs:

• kubelet: the node's agent. It asks the API server which pods belong on its node, and makes sure they're running.
• Container runtime (usually containerd): pulls images and starts containers, the same job Docker does on your laptop.
• A network plugin (CNI) gives each pod its own IP address, and kube-proxy routes traffic sent to a Service on to the right pods. The networking module covers both.

What really happens when you run kubectl apply

This is the path a single change takes through a cluster. Knowing it tells you where to look when something doesn't happen: each arrow is a place where things can get stuck.

Notice that kubectl apply returns as soon as the API server has stored your change, near the top of the diagram. Everything below it happens afterwards, without you. That's why "the command succeeded" and "the app is running" are two different questions, a point the Incident Triage module returns to.

On a managed service such as Amazon EKS, the control plane half of this diagram is run by the cloud provider and the worker nodes are yours. Kubernetes on Amazon EKS covers exactly where that line falls.

Look around a cluster

These read-only commands work on any cluster, including a local kind cluster:

# The worker nodes, with their Kubernetes version and internal IPs
kubectl get nodes -o wide

# Everything running in every namespace, including the system pods in kube-system
kubectl get pods -A

# On kind you can see the control plane itself: etcd, the API server,
# the scheduler, and the controller manager all run as pods in kube-system
kubectl get pods -n kube-system -o wide

# Every kind of object the API server knows about
kubectl api-resources

# Watch the reconciliation loop: delete a pod a Deployment owns,
# and a replacement appears within seconds
kubectl get pods -w

Key terms

• Desired state: what you've declared should exist.
• Controller: a loop that compares desired and actual state, and acts on any difference.
• API server: the only way in to the cluster, for people, tools, nodes, and controllers.
• etcd: the database that stores every object.
• Scheduler: chooses a node for each new pod, based on what the pod requests.
• kubelet: the agent on each node that starts and watches the pods assigned to it.

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.