Why pod is in init state with the error Permission denied on volume creation inside pod?
Permission issues on the storage partition may be one of the causes if the pod is stuck in the init state. To identify and fix this problem, do the following actions:
Check the logs:
- Describe the pod and see if there is any log related to the volume creation failed,
- kubectl describe pod -n
- if so, run the diagnostic tool to see if there are any noexec, nodev etc there
- cd /opt/UiPathAutomationSuite///Suppurt-Tools/diagnostics-tool
- ./diagnostics-report.sh
- validate the output like below
- If noexec, nodev etc Validation failed follow the below steps
Resolution steps:
Cause: Something (configuration management or something similar) likely added noexec , nodev, nosuid to /var mount, and this caused Longhorn and possibly other pods to not function.
Solution:
- Remove noexec, nodev, and nosuid for /var in /etc/fstab
- Drain all pods running on a node
- Restart the node
- Check if the aforementioned mount options are removed by running findmnt -l | grep '/var' -C 2
- Perform steps 1- 4 on the rest of the node