How to resolve VAR full space errors in Automation Suite?
Error Message: Pods in Error, ContainerCreating.
Error message: 'failed to create containerd task: failed to create container loggers: Failed to create and open log file: open /var/log/pods/smb-driver_csi-smb-controller-7b86bdf468-kxwgk_4b5905de-1308-438a-984d-efded102d6a9/csi-provisioner/478.log: no space left on device' .
Diagnosis:
- On the pods affected, run command kubectl -n (namespace of pod) describe pod (pod name). Observe the error messages here. If error message is similar to message: 'failed to create contained task: failed to create container loggers: Failed to create and open log file: open /var/log/pods/smb-driver_csi-smb-controller-7b86bdf468-kxwgk_4b5905de-1308-438a-984d-efded102d6a9/csi-provisioner/478.log: No space left on device'.
- Run df -h /var. (or on the partition that is saying has no space)
- This should return whether there is space or not.
- If space is not available, expand partition or clear some pod logs.
Root Cause: /var partition becomes full and pods are impacted.
Resolution:
- Run find /var/log/pods -type f -atime +10 -delete
This will delete the pod log folder and files and should clear up some space.