Upgrading It, Breaking Its Brain, and Putting It Back Together

Standing a cluster up once is one skill. Keeping it alive through the changes that happen to any piece of infrastructure over time — a version upgrade, a node that needs replacing, a datastore that needs recovering from a bad day — is a different, arguably more important one. This installment is about that second skill: taking the cluster from the last two posts and actually operating it.

Upgrading it without losing anything

A Kubernetes upgrade isn't one action, it's a sequence, and the sequence matters. The control plane goes first — a tool called kubeadm walks it through certificate renewal and a rolling restart of its own core components, one at a time, while the cluster keeps answering requests the whole way through.

The workers are where the real discipline is. Each one gets drained before it's touched — everything running on it gets rescheduled onto the other healthy nodes first — upgraded, and only then brought back into rotation:

kubectl drain k8s-w1 --ignore-daemonsets --delete-emptydir-data

Doing them one at a time, never both at once, is the entire point: the cluster never loses more than one node's worth of capacity at any moment, and a mistake on the first node is caught and fixed before it's repeated on the second.

One quiet software-supply-chain fact worth knowing: the tool that manages the upgrade (kubeadm) and the actual node agent it's upgrading (kubelet) are separate packages that don't upgrade each other automatically — you have to do both, deliberately, in the right order. Skip that and you end up with a management tool that thinks it upgraded something it never actually touched.

The part that actually matters: does the backup really work?

Every piece of state in a Kubernetes cluster — every deployment, every secret, every fact about what's supposed to be running where — lives in one place: etcd, a small distributed database that everything else in the cluster treats as the single source of truth. The API server, the scheduler, all of it, are essentially stateless front ends reading from and writing to that one store. Back up etcd correctly, and you've backed up the entire cluster's brain. Get the backup or the restore wrong, and no amount of YAML sitting in a git repo saves you.

So instead of just running snapshot save once and calling it proven, the test here was designed to actually demonstrate something:

etcdctl snapshot save /root/etcd-backups/before-canary.db
etcdutl snapshot status /root/etcd-backups/before-canary.db -w table

┌──────────┬──────────┬────────────┬────────────┬─────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │ VERSION │
├──────────┼──────────┼────────────┼────────────┼─────────┤
│ e7db6baa │  4437891 │        605 │      11 MB │   3.7.0 │
└──────────┴──────────┴────────────┴────────────┴─────────┘

Take that snapshot, then create a brand-new namespace after it — standing in for real work done since your last backup — then restore from the earlier snapshot and check whether the new namespace survived:

kubectl get ns canary
Error from server (NotFound): namespaces "canary" not found

It didn't. And that's the correct, if slightly uncomfortable, result.

Anything that happened after that instant is gone with it, which is precisely why how often you snapshot matters as much as whether you snapshot at all. Meanwhile, everything that existed before the snapshot — the actual applications running on the cluster — came back untouched, which is the other half of the proof: a correct restore, not a destructive one.

Getting there involved briefly and deliberately taking the control plane offline (moving its components' configuration out of the way so they stop, restoring the database underneath them into a fresh directory rather than overwriting the only copy of the current data, then putting the configuration back) — a sequence that's genuinely nerve-wracking to run the first time and much less so the second.

Adding a node, removing a node, and a bug that only a truly new node could find

Scaling a cluster up is the easy direction — provision a new machine, run the same preparation steps as the others, and join it in with a token generated fresh for the occasion (the one from the original setup expired within the first day, which is by design — long-lived join credentials sitting around is exactly the kind of thing you don't want).

That new node immediately found a real bug in the setup script that three earlier nodes had been silently hiding. crictl — the CLI you actually inspect containers with when there's no Docker installed — comes from a package called cri-tools, and cri-tools ships from the same Kubernetes community package repository as kubelet/kubeadm/kubectl, not from the operating system's own default one. The setup script installed it a step too early, before that repository had been configured. It had "worked" on the original three nodes only because that repository already happened to exist there from an earlier manual fix, weeks prior. A script that looks correct can still be wrong in a way that only shows up the next time someone starts from a truly blank machine — which is exactly why re-testing your own tooling against a fresh node, not just trusting that it worked once, is worth doing.

Taking a node back out cleanly is its own small discipline: drain it, remove it from the cluster, then run the reset command on the node itself — which prints its own warning about exactly what it won't do for you:

The reset process does not perform cleanup of CNI plugin configuration,
network filtering rules and kubeconfig files.

Worth checking that for real once rather than taking the warning's word for it:

ls /etc/cni/net.d/
10-calico.conflist  calico-kubeconfig

iptables-save | grep -c KUBE
145

Both still there. Harmless on a node about to be deleted entirely, but a genuinely different situation on a node being repurposed instead of retired — exactly the kind of gap that's easy to file away as "check for this" rather than actually forget about.

Where this leaves things

Between this post and the last two, the cluster has now been built from nothing, networked and stored data two different ways with no cloud provider involved, served its own container images over its own private registry, upgraded live, had its central datastore genuinely destroyed and recovered, and grown and shrunk a node. That's a complete operational lifecycle, not just a working demo — the exact difference between having deployed something once and being able to run it.

Two things earn themselves a follow-up: a deliberate round of breaking things on a timer — killing kubelet, filling a disk, corrupting containerd's config, each one diagnosed against the clock — and pushing the whole thing under real load to see where it actually gives. Both are squarely in reach now that the plumbing's proven end to end.