Kubernetes API Server Certificate Expired — Cluster Control Plane Down
Kubernetes cluster API server is unreachable because TLS certificates have expired. kubeconfig connections fail with x509 certificate has expired errors. Control plane components (scheduler, controller-manager, etcd) lose connectivity. Certificates generated by kubeadm default to 1-year expiry.
Indicators
- kubectl commands fail: x509: certificate has expired or is not valid yet
- kube-apiserver pod not starting or CrashLoopBackOff
- Nodes show NotReady in kubectl get nodes (when accessible)
- kubeadm certs check-expiration shows EXPIRED or < 7d remaining
- etcd health check fails: context deadline exceeded
Likely causes
- kubeadm-generated certificates reached 1-year default expiry
- Cluster time sync failure causing certificate validity window mismatch
- Certificate renewal job/process failed silently at renewal point
- Manual certificate rotation performed incorrectly
Diagnostic steps
-
From a control-plane node: kubeadm certs check-expiration — lists all cert expiry dates and flags expired certs
-
Check system time on control-plane nodes: date && timedatectl status — ensure NTP is synced and all nodes agree
-
Verify API server is running: crictl ps | grep kube-apiserver — if not running, check: journalctl -u kubelet -n 100
-
Check API server manifest for cert paths: cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep tls-cert
-
Attempt renewal: kubeadm certs renew all — renews all control-plane certs; requires root on control-plane node
-
After renewal, restart static pods: move manifests out of /etc/kubernetes/manifests/ then back in, or: systemctl restart kubelet
Resolution path
- Run kubeadm certs renew all on each control-plane node
- Restart kubelet to pick up renewed certs: systemctl restart kubelet
- Regenerate kubeconfig files: kubeadm init phase kubeconfig all
- Copy new admin.conf to ~/.kube/config to restore kubectl access
- Set up cert-manager or an automated renewal cron to prevent recurrence
Prevention
- Schedule annual certificate renewal before expiry: set calendar reminders at 11 months
- Use cert-manager for automated TLS rotation in managed clusters
- Monitor certificate expiry with Prometheus kube-state-metrics cert_expiration alerts
Tools
- kubeadm certs check-expiration / kubeadm certs renew all
- openssl x509 -in <cert> -text -noout | grep Validity
- crictl — container runtime interface CLI
- journalctl -u kubelet
- timedatectl / chronyc