居于 kubeadm部署的etcd备份恢复

前言

在 k8s运行过程中难免会遇到 etcd集群异常的情况。我们该如何做到备份以及恢复。

端口	作用
2379	提供 HTTP API 服务，供客户端交互
2380	和集群中其他节点通信

ETCD 备份

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt  --key=/etc/kubernetes/pki/etcd/server.key --endpoints=https://<IP>:2379 snapshot save ./etcd-snapshot-`date +%Y%m%d`.db

ETCD恢复

• 将备份文件传到所有etcd节点中

  使用 scp拷贝到各个机器·

• 逐台停止 kube-apiserver以及所有 etcd服务

  mv /etc/kubernetes/manifests/{etcd.yaml,kube-apiserver.yaml} /tmp/

• 备份etcd数据目录并留空

  rm -rf /var/lib/etcd

• 逐台执行命令去恢复

  ETCDCTL_API=3 etcdctl snapshot restore ./etcd-snapshot-20211009.db \
    --name <HOSTNAME> \
  #   --initial-cluster "<HOSTNAME>=https://<IP>:2380" \ 单节点则写单个
    --initial-cluster "<HOSTNAME>=https://<IP>:2380,<HOSTNAME>=https://<IP>:2380,<HOSTNAME>=https://<IP>:2380" \
    --initial-cluster-token etcd-cluster \
    --initial-advertise-peer-urls https://<IP>:2380 \ # 集群多个节点则往后太添加《,https://<IP>:2380》
    --data-dir=/var/lib/etcd/

• 逐台启动启动 api-service以及 etcd服务

  mv /tmp/{etcd.yaml,kube-apiserver.yaml} /etc/kubernetes/manifests/

• 检查etcd集群状态

  ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt  --key=/etc/kubernetes/pki/etcd/server.key --endpoints=https://<IP>:2379   member list

• kubelet检查

  kubectl get node
  kubectl get cs

ETCD操作

• 查询健康状态

  ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key --endpoints=https://<IP>:2379 endpoint health

• 查询所有key

  ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt  --key=/etc/kubernetes/pki/etcd/server.key --endpoints=https://<IP>:2379 get / --prefix --keys-only

• 查看成员

  ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt  --key=/etc/kubernetes/pki/etcd/server.key --endpoints=https://<IP>:2379   member list