How to resolve VAR full space errors in Automation Suite? Error message: 'failed to create containerd task: failed to create container loggers: Failed to create and open log file: open /var/log/pods/smb-driver_csi-smb-controller-7b86bdf468-kxwgk_4b5905de-1308-438a-984d-efded102d6a9/csi-provisioner/478.log: no space left on device' .
Error Presented: Pods in Error, ContainerCreating.
Diagnosis:
On the pods affected, run command kubectl -n (namespace of pod) describe pod (pod name). Observe the error messages here. If error message is similar to message: 'failed to create contained task: failed to create container loggers: Failed to create and open log file: open /var/log/pods/smb-driver_csi-smb-controller-7b86bdf468-kxwgk_4b5905de-1308-438a-984d-efded102d6a9/csi-provisioner/478.log: No space left on device'.
- Run df -h /var. (or on the partition that is saying has no space)
This should return whether there is space or not.
If space is not available, expand partition or clear some pod logs.
Root Cause: /var partition had become full and pods were impacted.
Resolution:
- Run find /var/log/pods -type f -atime +10 -delete
This will delete the pod log folder and files and should clear up some space.
Palaniyappan
Nguyen_Van_Luong1
Anil_G
mpraveen1902
Yoichi
ppr
Shiva_Nikhil
mkankatala
supermanPunch
anjasing