Kubernetes -- NFS作为动态存储创建pvc/pv kubernetesnfs

【Kubernetes -- NFS作为动态存储创建pvc/pv】NFS由于自身的问题，不常用于生产环境，这里仅作为demo展示动态存储的使用。生产环境可以使用ceph，rook-ceph来管理ceph存储。
假设已部署好NFS Server，这里演示如何在集群中部署动态存储与创建storageclass/pvc/pv。
1.创建独立的namespace

# kubectl create ns storage

2.创建rbac给serviceAccount赋权创建一个serviceAccount:

apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner namespace: storage

为serviceAccount赋权：

kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]

kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: storage roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io

3.部署Provisioner privisioner可以理解为底层存储的驱动，由privisioner管理底层存储。
privisioner以deploy方式部署了1个pod，pod内container指定了nfs的环境信息(包括name/ip/path等)，serviceAccountName=上一步创建的serviceAccount名称；

apiVersion: apps/v1 kind: Deployment metadata: labels: app: nfs-provisioner name: nfs-provisioner namespace: storage spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-provisioner template: metadata: labels: app: nfs-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest volumeMounts: - name: nfs-volume mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 178.104.163.63 - name: NFS_PATH value: /var/nfs volumes: - name: nfs-volume nfs: server: 178.104.163.63 path: /var/nfs

查看部署的pod:

# kubectl get pod -n storage NAMEREADYSTATUSRESTARTSAGE nfs-provisioner-778c655cbd-4twcz1/1Running019s

provisioner部署完毕后，观察pod的log；若有报错，需及时排查：

# kubectl logs nfs-provisioner-778c655cbd-4twcz -n storage I0330 08:08:27.9816521 leaderelection.go:185] attempting to acquire leader leasestorage/fuseim.pri-ifs... I0330 08:08:27.9929691 leaderelection.go:194] successfully acquired lease storage/fuseim.pri-ifs I0330 08:08:27.9930281 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776! I0330 08:08:27.9935041 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"storage", Name:"fuseim.pri-ifs", UID:"35414cb4-5e1c-45e4-9d0e-9b031e0c3df2", APIVersion:"v1", ResourceVersion:"2323405", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776 became leader I0330 08:08:28.0933431 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-provisioner-778c655cbd-4twcz_1bbd49e5-912f-11eb-a36b-fef168fb5776!

4.创建storageclass

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-sc namespace: storage provisioner: fuseim.pri/ifs# 这里的provisioner==上面env.PROVISIONER_NAME

查看storageclass:

# kubectl get sc -n storage NAMEPROVISIONERRECLAIMPOLICYVOLUMEBINDINGMODEALLOWVOLUMEEXPANSIONAGE nfs-scfuseim.pri/ifsDeleteImmediatefalse19s

5.storageclass创建pvc给pod使用创建pvc：

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myclaim namespace: storage spec: accessModes: - ReadWriteMany volumeMode: Filesystem resources: requests: storage: 1Gi storageClassName: nfs-sc//使用storageClass

查看自动创建的pv:

# kubectl get sc,pv,pvc -n storage NAMEPROVISIONERRECLAIMPOLICYVOLUMEBINDINGMODEALLOWVOLUMEEXPANSIONAGE storageclass.storage.k8s.io/nfs-scfuseim.pri/ifsDeleteImmediatefalse13mNAMECAPACITYACCESS MODESRECLAIM POLICYSTATUSCLAIMSTORAGECLASSREASONAGE persistentvolume/pvc-2fa56db1-fe02-4722-a9e9-d0dfad5659341GiRWXDeleteBoundstorage/myclaimnfs-sc3m12sNAMESTATUSVOLUMECAPACITYACCESS MODESSTORAGECLASSAGE persistentvolumeclaim/myclaimBoundpvc-2fa56db1-fe02-4722-a9e9-d0dfad5659341GiRWXnfs-sc3m12s

将pvc提供给pod使用：

kind: Pod apiVersion: v1 metadata: name: test-pod namespace: storage spec: containers: - name: test-pod image: nginx:1.15.2 volumeMounts: - name: nfs-pvc mountPath: /mnt/nginx volumes: - name: nfs-pvc persistentVolumeClaim: claimName: myclaim## claimName==上面创建的pvc的名称

pod创建完毕，到容器中查看挂载的目录：

# kubectl exec -it test-pod -n storage -- bash # df -h FilesystemSizeUsed Avail Use% Mounted on overlay100G6.2G94G7% / tmpfs64M064M0% /dev tmpfs3.0G03.0G0% /sys/fs/cgroup /dev/vda1100G6.2G94G7% /etc/hosts 178.104.163.63:/var/nfs/storage-myclaim-pvc-2fa56db1-fe02-4722-a9e9-d0dfad565934100G1.4G99G2% /mnt/nginx

6.storageclass为statefulset创建pvc/pv statefulset.spec指定storageClass，部署时会自动为其创建pvc、pv：

apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: nginx name: nginx namespace: storage spec: serviceName: "nginx"# 等于headless service的名称 replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx:1.15.2 imagePullPolicy: IfNotPresent name: nginx terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/nginx name: nfs-sc-volume restartPolicy: Always schedulerName: default-scheduler terminationGracePeriodSeconds: 30 volumeClaimTemplates:## 指定连接到sc去申请pvc - metadata: name: nfs-sc-volume spec: accessModes: - ReadWriteOnce resources: limits: storage: 2Gi requests: storage: 1Gi storageClassName: nfs-sc volumeMode: Filesystem

这里的statefulset创建了2个pod，为每个pod都绑定了一个pvc，2个pod使用独立的存储卷:

# kubectl get sc,pvc,pv -n storage NAMEPROVISIONERRECLAIMPOLICYVOLUMEBINDINGMODEALLOWVOLUMEEXPANSIONAGE storageclass.storage.k8s.io/nfs-scfuseim.pri/ifsDeleteImmediatefalse4m58sNAMESTATUSVOLUMECAPACITYACCESS MODESSTORAGECLASSAGE persistentvolumeclaim/nfs-sc-volume-nginx-0Boundpvc-63389a42-a00d-4b34-bd51-4542cebb42aa1GiRWOnfs-sc75s persistentvolumeclaim/nfs-sc-volume-nginx-1Boundpvc-e01d4f5c-004b-4618-94d6-9556612cd1981GiRWOnfs-sc70sNAMECAPACITYACCESS MODESRECLAIM POLICYSTATUSCLAIMSTORAGECLASSREASONAGE persistentvolume/pvc-63389a42-a00d-4b34-bd51-4542cebb42aa1GiRWODeleteBoundstorage/nfs-sc-volume-nginx-0nfs-sc75s persistentvolume/pvc-e01d4f5c-004b-4618-94d6-9556612cd1981GiRWODeleteBoundstorage/nfs-sc-volume-nginx-1nfs-sc70s