一、背景

当node节点进行如打补丁、操作系统升级等操作时，需停机维护，这就涉及pod驱逐迁移，本文将详细介绍node节点维护的整个过程。

二、pdb简介

pdb为poddisruptionbudgets缩写，意为主动驱逐保护；
没有pdb。当进行节点维护时，如果某个服务的多个pod在该节点上，则节点的停机可能会造成服务中断或者服务降级。举个例子，某服务有5个pod，最低3个pod能保证服务质量，否则会造成响应慢等影响，此时该服务的4个pod在node01上，如果对node01进行停机维护，此时只有1个pod能正常对外服务，在node01的4个pod迁移过程中，就会影响该服务正常响应；
pdb能保证应用在节点维护时不低于一定数量的pod运行，从而保持服务质量；

三、准备工作

1.新建pod

[[email protected] ~]# more nginx-master.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-master
spec:
  replicas: 10 
  template:
    metadata:
      labels:
        app: nginx
    spec:
      restartPolicy: Always
      containers:
      - name: nginx
        image: nginx:latest
[[email protected] ~]# kubectl apply -f nginx-master.yml 
deployment.extensions/nginx-master created
[[email protected] ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          28s   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-69jn6   1/1     Running   0          28s   10.244.2.206   node02   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          28s   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          28s   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-fxsjd   1/1     Running   0          28s   10.244.2.204   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          28s   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          28s   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pcznk   1/1     Running   0          28s   10.244.2.203   node02   <none>           <none>
nginx-master-9d4cf4f77-px98b   1/1     Running   0          28s   10.244.2.205   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          28s   10.244.1.220   node01   <none>

新建pod，镜像为最新版的nginx，deployment为nginx-master，数量为10。可以看到10个pod分布在node01、node02和master 3台不同主机上。

2.新建pdb

[[email protected] ~]# more pdb-nginx.yaml 
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: pdb-nginx
spec:
  minAvailable: 9
  selector:
    matchLabels:
      app: nginx
[[email protected] ~]# kubectl apply -f pdb-nginx.yaml 
poddisruptionbudget.policy/pdb-nginx created
[[email protected] ~]# kubectl get pdb
NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
pdb-nginx   9               N/A               1                     8s

新建pdb pdb-nginx，Label Selector和deployment一样都为app: nginx，minAvailable: 9意为存活的nginx pod至少为9个。

四、节点维护

本文以节点node02维护为例介绍。

1.设置节点不可调度

[[email protected] ~]# kubectl cordon node02
node/node02 cordoned
[[email protected] ~]# kubectl get node
NAME     STATUS                     ROLES    AGE    VERSION
master   Ready                      master   184d   v1.14.2
node01   Ready                      <none>   183d   v1.14.2
node02   Ready,SchedulingDisabled   <none>   182d   v1.14.2
[[email protected] ~]# kubectl get po -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
nginx-master-9d4cf4f77-47vfj   1/1     Running   0          30m   10.244.0.129   master   <none>           <none>
nginx-master-9d4cf4f77-69jn6   1/1     Running   0          30m   10.244.2.206   node02   <none>           <none>
nginx-master-9d4cf4f77-6drhg   1/1     Running   0          30m   10.244.1.218   node01   <none>           <none>
nginx-master-9d4cf4f77-b7zfd   1/1     Running   0          30m   10.244.1.219   node01   <none>           <none>
nginx-master-9d4cf4f77-fxsjd   1/1     Running   0          30m   10.244.2.204   node02   <none>           <none>
nginx-master-9d4cf4f77-ktnvk   1/1     Running   0          30m   10.244.0.128   master   <none>           <none>
nginx-master-9d4cf4f77-mzrx7   1/1     Running   0          30m   10.244.1.217   node01   <none>           <none>
nginx-master-9d4cf4f77-pcznk   1/1     Running   0          30m   10.244.2.203   node02   <none>           <none>
nginx-master-9d4cf4f77-px98b   1/1     Running   0          30m   10.244.2.205   node02   <none>           <none>
nginx-master-9d4cf4f77-wtcwt   1/1     Running   0          30m   10.244.1.220   node01   <none>

设置node02不可调度，查看各节点状态，发现node02为SchedulingDisabled，此时master不会将新的pod调度到该节点上，但是node02上pod还是正常运行。

2.驱逐节点上的pod

[[email protected] ~]# kubectl drain node02 --delete-local-data --ignore-daemonsets --force 
node/node02 already cordoned

参数说明：

--delete-local-data 即使pod使用了emptyDir也删除
--ignore-daemonsets 忽略deamonset控制器的pod，如果不忽略，deamonset控制器控制的pod被删除后可能马上又在此节点上启动起来,会成为死循环；
--force 不加force参数只会删除该NODE上由ReplicationController, ReplicaSet, DaemonSet,StatefulSet or Job创建的Pod，加了后还会删除'裸奔的pod'(没有绑定到任何replication controller)

3.维护结束

[[email protected] ~]# kubectl uncordon node02
node/node02 uncordoned
[[email protected] ~]# kubectl get nodes      
NAME     STATUS   ROLES    AGE    VERSION
master   Ready    master   184d   v1.14.2
node01   Ready    <none>   183d   v1.14.2
node02   Ready    <none>   183d   v1.14.2

维护结束，重新将node02节点置为可调度状态。

节点删除

1.删除节点

实际运维过程中可能会删除某个node节点，本文还是以node02为例，介绍如果删除节点。

[[email protected] ~]# kubectl cordon node02
[[email protected] ~]# kubectl drain node02 --delete-local-data --ignore-daemonsets --force 
[[email protected] ~]# kubectl delete node node02

[[email protected] ~]# kubeadm reset

重新加入

[[email protected] ~]# kubeadm token create --print-join-command
kubeadm join 172.27.9.131:6443 --token kpz40z.tuxb4t4m1q37vwl1     --discovery-token-ca-cert-hash sha256:5f656ae26b5e7d4641a979cbfdffeb7845cc5962bbfcd1d5435f00a25c02ea50

[[email protected] ~]# kubeadm join 172.27.9.131:6443 --token svrip0.lajrfl4jgal0ul6i     --discovery-token-ca-cert-hash sha256:5f656ae26b5e7d4641a979cbfdffeb7845cc5962bbfcd1d5435f00a25c02ea50