Kubernetes Storage and Upgrades

Storage and upgrades are where Kubernetes stops being purely declarative and starts touching durable state. Treat both as operational workflows with backups, sequencing, and rollback plans.

This page was reviewed against the Kubernetes v1.36 documentation in May 2026. As of May 24, 2026, the upstream releases page lists v1.36.1 as the latest v1.36 patch release and lists v1.36, v1.35, and v1.34 as the maintained minor release branches.

Storage Building Blocks

PVs and PVCs decouple application YAML from infrastructure implementation. A developer asks for “20Gi fast storage”; the platform decides which backend provides it.

CSI

CSI is the Container Storage Interface. Kubernetes uses it to integrate storage systems without baking vendor-specific volume plugins into core Kubernetes.

Typical CSI components:

• Controller plugin: provisions, deletes, attaches, detaches, expands, and snapshots volumes.
• Node plugin: stages and publishes volumes on nodes so Pods can mount them.
• External sidecars: provisioner, attacher, resizer, snapshotter, health monitor depending on driver features.

Operational lesson: CSI driver compatibility matters. Kubernetes may support an API, but the driver and sidecars must support the exact feature and version combination.

CSI sidecar compatibility examples:

Access Modes

Do not assume ReadWriteMany is available. Many block storage systems only support single-node attachment.

StatefulSets

StatefulSets provide stable identity:

• predictable Pod names,
• ordered rollout behavior,
• stable PVCs through volumeClaimTemplates,
• stable DNS when paired with a headless Service.

They do not make the application safe. PostgreSQL, Kafka, etcd, and similar systems still need application-level replication, quorum, backup, and recovery logic.

StatefulSet volume migration shape:

flowchart LR
  OldPVC[Old PVC / StorageClass] --> Snapshot[VolumeSnapshot or app backup]
  Snapshot --> NewPVC[New PVC / target StorageClass]
  NewPVC --> Canary[Canary Pod restore validation]
  Canary --> Cutover[Scale down or switchover]
  Cutover --> NewSTS[StatefulSet uses migrated volume]

Prefer application-native replication or backup/restore when the storage backend cannot safely clone volumes across classes, zones, or encryption domains. Kubernetes can move objects; it cannot prove database-level consistency by itself.

Reclaim Policies

Reclaim policy controls what happens to the PV after its PVC is deleted:

• Delete: delete the backing volume through the provisioner.
• Retain: keep the backing volume for manual recovery or cleanup.

For databases, understand reclaim policy before running cleanup automation. Accidentally deleting PVCs with a Delete policy can delete the underlying disk.

Kubernetes Upgrades

The safe high-level upgrade sequence is:

1. Read release notes, deprecations, API removals, and add-on compatibility.
2. Back up etcd and application data.
3. Check version skew policy for API server, kubelet, kubeadm, kubectl, controller manager, scheduler, and add-ons.
4. Upgrade one primary control plane node.
5. Upgrade additional control plane nodes.
6. Upgrade worker nodes in controlled batches.
7. Upgrade CNI, CSI, CoreDNS, ingress/gateway controllers, and cloud controllers as required by their compatibility matrices.
8. Verify workloads, DNS, Service routing, admission webhooks, storage attach/mount, and node readiness.

For kubeadm clusters, skipping minor versions is unsupported. Move one minor at a time and stay on supported release lines.

Upgrade Inventory

Before selecting a target version, capture what the cluster is actually running:

kubectl version
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get apiservices
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration
kubectl get pdb -A
kubectl get deployment,daemonset,statefulset -A
kubeadm upgrade plan

Inventory these owners:

Run deprecation checks before the upgrade, not during it. A removed API can break a controller even if the currently stored objects still appear healthy.

Version Skew and Order

Kubernetes version skew policy drives the safe order. The API server moves first, and node components must not be newer than the API server they depend on.

flowchart LR
  A[Release notes and deprecation scan] --> B[etcd and app backups]
  B --> C[First control plane node]
  C --> D[Remaining control plane nodes]
  D --> E[CNI, CSI, CoreDNS, kube-proxy, controllers]
  E --> F[Worker node batches]
  F --> G[Post-upgrade validation]

In an HA control plane, upgrade API servers one at a time and remember that temporary skew narrows what node and controller versions are valid. Do not start upgrading nodes until the control plane is healthy at the target minor version.

kubeadm Control Plane Workflow

For kubeadm-managed clusters, use the upstream kubeadm flow, not ad hoc static Pod edits:

1. Upgrade package repositories or node images for the target minor and patch version.
2. On the first control plane node, run kubeadm upgrade plan.
3. Run kubeadm upgrade apply <target-version> on the first control plane node.
4. Upgrade and restart that node’s kubelet and kubectl packages according to your OS packaging flow.
5. Drain, upgrade, restart, and uncordon that control plane node when the kubelet changes.
6. On each additional control plane node, run kubeadm upgrade node instead of kubeadm upgrade apply.
7. Upgrade CNI and other add-ons according to their own compatibility matrix.
8. Confirm all control plane Pods, nodes, admission webhooks, CoreDNS, Services, and storage paths are healthy before worker upgrades.

Since Kubernetes v1.28, kubeadm waits until all control plane instances are upgraded before upgrading add-ons such as CoreDNS and kube-proxy by default. Still read the kubeadm output and release notes carefully; add-on behavior can be version- and distribution-specific.

Worker Node Workflow

Upgrade workers in small batches:

1. Confirm there is enough spare capacity for evicted workloads.
2. Cordon the node.
3. Drain it while respecting PodDisruptionBudgets and application maintenance rules.
4. Upgrade kubelet, kubeadm, container runtime packages, OS packages, and node image as applicable.
5. Restart kubelet and confirm the node returns Ready.
6. Uncordon the node.
7. Watch workload readiness, Service endpoints, DNS, storage mounts, and application error rates before the next batch.

Do not use --disable-eviction or broad --force drains as normal procedure. If a drain hangs, inspect PDBs, local storage, DaemonSets, finalizers, and stuck terminating Pods. For stateful workloads, the database or queue runbook owns safe disruption timing; Kubernetes only performs the eviction mechanics.

Add-On and Platform Upgrades

Kubernetes upgrades are incomplete until add-ons are compatible:

Managed Kubernetes changes the command surface, not the responsibility. Cloud providers may upgrade the control plane for you, but you still own release-note review, add-on compatibility, node pool rollouts, PDBs, stateful workload disruption, and post-upgrade validation.

Upgrade Risks

Post-Upgrade Validation

Check platform behavior before declaring the upgrade complete:

kubectl get --raw /readyz?verbose
kubectl get nodes
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp
kubectl -n kube-system rollout status deploy/coredns
kubectl get endpointslice -A
kubectl get volumeattachment

Validate:

• API writes and watches still work.
• Admission webhooks are reachable and not blocking routine deploys.
• New Pods get IPs and can resolve cluster DNS.
• Services and EndpointSlices update when Pods roll.
• Ingress, Gateway, and load balancer controllers still program traffic.
• PVC attach, mount, resize, and snapshot workflows still work.
• Stateful applications meet quorum, replication, and backup requirements.

Rollback and Recovery

Kubernetes rollback is constrained. You can usually roll back a node image or add-on quickly, but control plane downgrades are not a casual operation and may be unsupported after storage migrations or API changes. Plan an abort point before each phase:

Commands

kubectl get pv,pvc,storageclass,volumesnapshotclass
kubectl describe pvc <claim>
kubectl get volumeattachment
kubectl version
kubectl get nodes -o wide
kubectl get apiservices
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration
kubectl get pdb -A
kubeadm upgrade plan
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <node>

Study Cards

References

• Kubernetes Persistent Volumes
• Kubernetes Storage
• Upgrading kubeadm clusters
• Kubernetes version skew policy
• Kubernetes releases