Bind Mount

[chadlangston]

The bind mount does not appear to succeed on my K8S deployment. The mount pod is working, but the storage volume in my app is not bind mounting to the mount pod volume. The bind mount log message from the CSI node pod appears to be successful and there are no errors on the nodes. My kubernetes cluster is a microk8s cluster. I installed bindfs on the nodes thinking that it may be required, but that didn't do the trick. Any thoughts would be greatly appreciated :)

Specifically, I see

juicefs.go:165] BindTarget: binding /jfs/juicefs-pv-yklsss at /var/snap/microk8s/common/var/lib/kubelet/pods/1521f287-94a4-46b7-a943-4867f5267ae3/volumes/kubernetes.io~csi/juicefs-pv/mount

and

node.go:134] NodePublishVolume: mounted juicefs-pv at /var/snap/microk8s/common/var/lib/kubelet/pods/1521f287-94a4-46b7-a943-4867f5267ae3/volumes/kubernetes.io~csi/juicefs-pv/mount with options []

in the csi pod logs. And I see out.txt here /var/snap/microk8s/common/var/lib/kubelet/pods/1521f287-94a4-46b7-a943-4867f5267ae3/volumes/kubernetes.io~csi/juicefs-pv/mount, but I don't see out.txt here /jfs/juicefs-pv-yklsss

However, if I create files in /jfs/juicefs-pv-yklsss of the mount pod they are saved to my cloud storage and they persist the next time I mount the same juicefs backend.

[zwwhdls]

Hi @chadddddd , can you check kubelet-dir mountVolume of juicefs-csi-node-xxx ?

kubectl -n kube-system get po juicefs-csi-node-xxx -oyaml | grep kubelet-dir -A 5 -B 5

Replace juicefs-csi-node-xxx with csi node pod name in your cluster.

If kubelet-dir not mounted in csi node, bind path will not work in app container.

[chadlangston]

Thank you! Here is the output:

kubectl -n kube-system get po juicefs-csi-node-9px62 -o yaml | grep kubelet-dir -A 5 -B 5
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/kubelet
      mountPropagation: Bidirectional
      name: kubelet-dir
    - mountPath: /csi
      name: plugin-dir
    - mountPath: /dev
      name: device-dir
    - mountPath: /jfs
--
    operator: Exists
  volumes:
  - hostPath:
      path: /var/lib/kubelet
      type: Directory
    name: kubelet-dir
  - hostPath:
      path: /var/lib/kubelet/csi-plugins/csi.juicefs.com/
      type: DirectoryOrCreate
    name: plugin-dir
  - hostPath:

And here are my helm values

# helm chart values
storageClasses:
- name: juicefs-b2-sc
  enabled: true
  reclaimPolicy: Delete
  kubeletDir: /var/snap/microk8s/common/var/lib/kubelet

The microk8s nodes have a /var/lib/kublet symlink to this kubeletDir. I tried installing jfs with and without this value set and the results are the same.

[zwwhdls]

kubeletDir in values.yaml is not under storageClasses, refering to https://github.com/juicedata/charts/blob/main/charts/juicefs-csi-driver/values.yaml#L34. And it seems not work in your cluster. kubelet-dir in csi node should be what you set. Can you check it and have a retry?