Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Shutdown node and wait some minutes to start it again, the pod stucks in Terminating state #149

Open
MingZhang-YBPS opened this issue Dec 11, 2024 · 1 comment

Comments

@MingZhang-YBPS
Copy link

MingZhang-YBPS commented Dec 11, 2024

Steps to reproduce

  1. create a deployment which uses csi-s3 to provision filesystem PV
    image
  2. then shutdown the node ltaucenw8ridob54 which hosts the first pod nginx-deployment-7d94b7b565-2rrqr
  3. after a while, the pod nginx-deployment-7d94b7b565-2rrqr goes to Terminating state, a new pod is launched on another node xalp9fb32t4idmcz. This is expected behavior
    image
  4. then start node ltaucenw8ridob54, I am expecting the terminating pod nginx-deployment-7d94b7b565-2rrqr should be removing automatically in short time. But it stays Terminating state even after 2 hours. The csi-s3 log as below
    image

Root cause
I think it is because the origin pod mount path disappear, so the umount command fails which blocks NodeUnpublishVolume process. CSI need to implement some error handling here
image

@MingZhang-YBPS
Copy link
Author

A similar issue #117

@MingZhang-YBPS MingZhang-YBPS changed the title Shutdown node and wait some minutes to start it again, the pod using csi-s3 stucks in Terminating state [Bug] Shutdown node and wait some minutes to start it again, the pod using csi-s3 stucks in Terminating state Dec 18, 2024
@MingZhang-YBPS MingZhang-YBPS changed the title [Bug] Shutdown node and wait some minutes to start it again, the pod using csi-s3 stucks in Terminating state [Bug] Shutdown node and wait some minutes to start it again, the pod stucks in Terminating state Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant