K8s NFS Provisioner

前置条件

使用 openfiler 作为 NFS

部署好 NFS Server，此处使用 openfiler 搭建（share 需要开启 no_root_squash）。

在 worker 上安装 nfs-utils 进行测试：

mount -t nfs 10.10.50.7:/mnt/vg/volume/nfs /mnt/

自建 NFS

# 所有服务端节点安装nfs
yum -y install nfs-utils
systemctl enable nfs-server rpcbind --now
# 创建nfs共享目录、授权
mkdir -p /root/nfs && chmod -R 777 /root/nfs
# 写入exports
cat > /etc/exports << EOF
/root/nfs 10.0.0.0/8(rw,sync,no_root_squash)
EOF
 
systemctl reload nfs-server

在 worker 上安装 nfs-utils 进行测试：

mount -t nfs 10.10.50.16:/root/nfs /mnt/

安装 NFS Provisioner（Helm 方式）

kubectl create ns nfs-provisioner

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --untar
helm install nfs-subdir-external-provisioner ./nfs-subdir-external-provisioner \
    --set nfs.server=10.10.50.7 \
    --set nfs.path=/mnt/vg/volume/nfs/k8s \
    --namespace nfs-provisioner

查看状态和日志：

[root@k8s-m01 ~]# kubectl -n nfs-provisioner get po
NAME                                               READY   STATUS    RESTARTS   AGE
nfs-subdir-external-provisioner-66b4f64688-dshzl   1/1     Running   0          57s

[root@k8s-m01 ~]# kubectl -n nfs-provisioner logs -f nfs-subdir-external-provisioner-66b4f64688-dshzl
I1204 05:13:15.660145       1 leaderelection.go:242] attempting to acquire leader lease  nfs-provisioner/cluster.local-nfs-subdir-external-provisioner...
I1204 05:13:15.668792       1 leaderelection.go:252] successfully acquired lease nfs-provisioner/cluster.local-nfs-subdir-external-provisioner
I1204 05:13:15.668848       1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"nfs-provisioner", Name:"cluster.local-nfs-subdir-external-provisioner", UID:"17e3e1bf-7295-4852-851e-8228ece8f867", APIVersion:"v1", ResourceVersion:"26068497", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' nfs-subdir-external-provisioner-66b4f64688-dshzl_d609688e-6108-45fa-aa49-74aeb1028b51 became leader
I1204 05:13:15.668922       1 controller.go:820] Starting provisioner controller cluster.local/nfs-subdir-external-provisioner_nfs-subdir-external-provisioner-66b4f64688-dshzl_d609688e-6108-45fa-aa49-74aeb1028b51!
I1204 05:13:15.769072       1 controller.go:869] Started provisioner controller cluster.local/nfs-subdir-external-provisioner_nfs-subdir-external-provisioner-66b4f64688-dshzl_d609688e-6108-45fa-aa49-74aeb1028b51!

查看 storage class：

[root@k8s-m01 ~]# kubectl get sc
NAME         PROVISIONER                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client   cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true                   7m51s

测试

cat > test-pvc.yaml <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
---
kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: busybox:stable
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim
EOF

kubectl apply -f test-pvc.yaml

检查 PVC 和 PV 状态：

[root@k8s-m01 ~]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-claim   Bound    pvc-54eaf0b5-0ef3-43e3-94f7-eb5a1155f028   1Mi        RWX            nfs-client     4s
[root@k8s-m01 ~]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE
pvc-54eaf0b5-0ef3-43e3-94f7-eb5a1155f028   1Mi        RWX            Delete           Bound    default/test-claim   nfs-client              7s