如何彻底删除EKS中一直卡在Terminating的Namespace kubernetes

作者： SRE运维博客
博客地址： https://www.cnsre.cn/
文章地址： https://www.cnsre.cn/posts/220107906441/
相关话题： https://www.cnsre.cn/tags/eks/

介绍 【如何彻底删除EKS中一直卡在Terminating的Namespace】最近一直在玩EKS（Elastic Kubernetes Service -- Amazon EKS）和 kubesphere。因为之前没有使用过EKS 和 kubesphere 所以这个过程也是一个试错的过程，在我使用 kubesphere 的时候发现有一个日志服务，在好奇心的驱使下，我创建了它。
在我创建了日志服务（KubeSphere Logging System）以后，我发现我并不想使用它。（可能我只是想看看它到底是什么吧。）强迫症我的，就想把它给删除掉。于是我在我的 EKS 中对他进行了强制删除 kubectl delete ns kubesphere-logging-system --force --grace-period=0
让人尴尬的是，这个 Namespace 并没有立马删除，我自我安慰到，可能 Namespace 下边有其他没有删除的资源在等待删除，我在等等。。。
过了半个小时，我去查看的时候

[root@ip-10-0-100-206 ~]# kubectl getnskubesphere-logging-system NAMESTATUSAGE kubesphere-logging-systemTerminating6d19h

它好像这地卡在了 Terminating 的状态。
我试着寻找解决方法，http://github.com/kubernetes/... 但是这中方法要通过 API 才可以实现。EKS 是托管在 AWS 中的，我根本没有办法去操作eks的后台。
我开始后悔我为什么要去安装它，我本可以不折腾的。但是回想如果我不去安装它的话，我肯定会像想在这样难过，因为我无法了解它（KubeSphere Logging System）到底是什么样的。
终于我找到了 commented https://github.com/kubernetes...
如何彻底删除 namespace 获取namespace的详情信息并转为json

kubectl get namespace kubesphere-logging-system -o json > kubesphere-logging-system.json

打开 json 文件编辑

{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "creationTimestamp": "2021-12-31T05:03:58Z", "deletionTimestamp": "2022-01-05T08:05:40Z", "labels": { "kubesphere.io/namespace": "kubesphere-logging-system", "kubesphere.io/workspace": "system-workspace" }, "managedFields": [ { "apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { ".": {}, "f:kubesphere.io/namespace": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": { ".": {}, "f:apiVersion": {}, "f:blockOwnerDeletion": {}, "f:controller": {}, "f:kind": {}, "f:name": {}, "f:uid": {} } } } }, "manager": "controller-manager", "operation": "Update", "time": "2021-12-31T05:04:01Z" }, { "apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:kubesphere.io/workspace": {} } }, "f:status": { "f:phase": {} } }, "manager": "kubectl", "operation": "Update", "time": "2021-12-31T05:04:01Z" }, { "apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": { "f:status": { "f:conditions": { ".": {}, "k:{\"type\":\"NamespaceContentRemaining\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceDeletionContentFailure\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceFinalizersRemaining\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} } } } }, "manager": "kube-controller-manager", "operation": "Update", "time": "2022-01-05T08:05:47Z" } ], "name": "kubesphere-logging-system", "ownerReferences": [ { "apiVersion": "tenant.kubesphere.io/v1alpha1", "blockOwnerDeletion": true, "controller": true, "kind": "Workspace", "name": "system-workspace", "uid": "6d535470-2592-4f3c-a155-eabc362c339d" } ], "resourceVersion": "7376520", "uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e" }, "spec": { "finalizers": [ "kubernetes" # 将此行删除 ] }, "status": { "conditions": [ { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "All resources successfully discovered", "reason": "ResourcesDiscovered", "status": "False", "type": "NamespaceDeletionDiscoveryFailure" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "All legacy kube types successfully parsed", "reason": "ParsedGroupVersions", "status": "False", "type": "NamespaceDeletionGroupVersionParsingFailure" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "All content successfully deleted, may be waiting on finalization", "reason": "ContentDeleted", "status": "False", "type": "NamespaceDeletionContentFailure" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances", "reason": "SomeResourcesRemain", "status": "True", "type": "NamespaceContentRemaining" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances", "reason": "SomeFinalizersRemain", "status": "True", "type": "NamespaceFinalizersRemaining" } ], "phase": "Terminating" } }

找到 spec 将 finalizers 下的 kubernetes 删除
具体如下

{ "apiVersion": "v1", "kind": "Namespace", "metadata": { "creationTimestamp": "2021-12-31T05:03:58Z", "deletionTimestamp": "2022-01-05T08:05:40Z", "labels": { "kubesphere.io/namespace": "kubesphere-logging-system", "kubesphere.io/workspace": "system-workspace" }, "managedFields": [ { "apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { ".": {}, "f:kubesphere.io/namespace": {} }, "f:ownerReferences": { ".": {}, "k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": { ".": {}, "f:apiVersion": {}, "f:blockOwnerDeletion": {}, "f:controller": {}, "f:kind": {}, "f:name": {}, "f:uid": {} } } } }, "manager": "controller-manager", "operation": "Update", "time": "2021-12-31T05:04:01Z" }, { "apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": { "f:metadata": { "f:labels": { "f:kubesphere.io/workspace": {} } }, "f:status": { "f:phase": {} } }, "manager": "kubectl", "operation": "Update", "time": "2021-12-31T05:04:01Z" }, { "apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": { "f:status": { "f:conditions": { ".": {}, "k:{\"type\":\"NamespaceContentRemaining\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceDeletionContentFailure\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} }, "k:{\"type\":\"NamespaceFinalizersRemaining\"}": { ".": {}, "f:lastTransitionTime": {}, "f:message": {}, "f:reason": {}, "f:status": {}, "f:type": {} } } } }, "manager": "kube-controller-manager", "operation": "Update", "time": "2022-01-05T08:05:47Z" } ], "name": "kubesphere-logging-system", "ownerReferences": [ { "apiVersion": "tenant.kubesphere.io/v1alpha1", "blockOwnerDeletion": true, "controller": true, "kind": "Workspace", "name": "system-workspace", "uid": "6d535470-2592-4f3c-a155-eabc362c339d" } ], "resourceVersion": "7376520", "uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e" }, "spec": { "finalizers": [ ] }, "status": { "conditions": [ { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "All resources successfully discovered", "reason": "ResourcesDiscovered", "status": "False", "type": "NamespaceDeletionDiscoveryFailure" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "All legacy kube types successfully parsed", "reason": "ParsedGroupVersions", "status": "False", "type": "NamespaceDeletionGroupVersionParsingFailure" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "All content successfully deleted, may be waiting on finalization", "reason": "ContentDeleted", "status": "False", "type": "NamespaceDeletionContentFailure" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances", "reason": "SomeResourcesRemain", "status": "True", "type": "NamespaceContentRemaining" }, { "lastTransitionTime": "2022-01-05T08:05:47Z", "message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances", "reason": "SomeFinalizersRemain", "status": "True", "type": "NamespaceFinalizersRemaining" } ], "phase": "Terminating" } }

执行清理命令
现在我们只需要一条命令就可以彻底删除这个 Namespace

kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json

执行完以后，你需要等待一会
再次执行命令检查 namespaces

kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json

最后的检查

kubectl getnskubesphere-logging-system Error from server (NotFound): namespaces "kubesphere-logging-system" not found[root@ip-10-0-100-206 ~]# kubectl getns NAMESTATUSAGE defaultActive23d kubesphere-controls-systemActive9d kubesphere-devops-systemActive9d kubesphere-devops-workerActive16h kubesphere-monitoring-federatedActive9d kubesphere-monitoring-systemActive9d kubesphere-sample-devActive8d kubesphere-systemActive9d

在此查看的时候，它已经不存在了。