kubernetes:pod无法删除 kubernetes:pod无法删除

kubernetes:v1.12.2
docker:18.06.1-ce
helm:v2.12.0

通过helm删除服务后，pods无法删除:

~]#helm del--purge simp-ser release "simp-ser" deleted//删除成功 ~]#helm install simp-ser --name=simp-ser Error: release simp-ser failed: object is being deleted: persistentvolumes "simp-ser" already exists //创建时报错，pvc已经存在

【kubernetes:pod无法删除】突如其来的报错让我有点懵，因为定义的这个pvc是跟随helm服务一起删除，我赶紧去看了下pvc服务:

~]#kubectl get pvc simp-serTerminatingsimp-ser300MiRWO,RWX3d21h//正在删除

pvc正在删除中，但是一直没有删除成功，因为关联了pod，我又查看了相关pod：

~]# kubectl get pods -o wide simp-ser-6b8f54bb64-tdxlw1/1Terminating03d21h10.244.1.71k8s.node1//也是正在删除 ~]#kubectl describe pods simp-ser-6b8f54bb64-tdxlw ... Container ID:docker://70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1 ... WarningFailedKillPod13m (x324 over 21h)kubelet, k8s.node1error killing pod: [failed to "KillContainer" for "simp-ser" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded" , failed to "KillPodSandbox" for "0e9a9fa9-6328-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded" ... ] //pod内的容器是70494faefcf...

看的出来是问题出在了node节点上，继续查看node的状态：

// 查看node是否正常 ~]# kubectl get node k8s.node1Ready146dv1.12.2 //切换到node节点查看kubelet 状态 ~]#systemctl status kubelet ... Active: active (running) since 五 2019-03-29 10:07:30 CST; 3 weeks 5 days ago.//kubelet状态也是正常的 ... //查看kubelet日志 ~]# journalctl -ukubelet -f ··· 4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.1557882147 kuberuntime_manager.go:810] Failed to stop sandbox {"docker" "ffc1eccb5b0ba196fe86f9987632194ee3c2e63372adb2d283a05ac0c2e68f4d"} 4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.1559342147 kubelet.go:1551] error killing pod: [failed to "KillContainer" for "simp-layui" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded" 4月 24 11:08:48 k8s.node1 kubelet[2147]: , failed to "KillPodSandbox" for "c5cae9f9-6321-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded" 4月 24 11:08:48 k8s.node1 kubelet[2147]: ] ··· //

既然kubelet没有问题，那再去看看pod中的容器：

~]# docker ps 70494faefcf3f4b4fccc65bb"sh -c 'java -jar /o…"3 days agoUp 3 daysk8s_simp-ser_simp-ser-6b8f54bb64-tdxlw_jx_0e9a9fa9-6328-11e9-bd84-000c2975aae4_0 // 容器还在运行 //查看下docker日志 ~]#journalctl -u docker -f ··· msg="Container 70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1 failed to exit within 30 seconds of signal 15 - using the force" ···//docker 无法 kill 掉容器