Pods Stuck In ContainerCreating ContainerConfigError In Automation Suite

News Knowledge Base
1

How to resolve VAR full space errors in Automation Suite?

Error Message: Pods in Error, ContainerCreating.

Error message: 'failed to create containerd task: failed to create container loggers: Failed to create and open log file: open /var/log/pods/smb-driver_csi-smb-controller-7b86bdf468-kxwgk_4b5905de-1308-438a-984d-efded102d6a9/csi-provisioner/478.log: no space left on device' .


Diagnosis:

  1. On the pods affected, run command kubectl -n (namespace of pod) describe pod (pod name). Observe the error messages here. If error message is similar to message: 'failed to create contained task: failed to create container loggers: Failed to create and open log file: open /var/log/pods/smb-driver_csi-smb-controller-7b86bdf468-kxwgk_4b5905de-1308-438a-984d-efded102d6a9/csi-provisioner/478.log: No space left on device'.
  • Run df -h /var. (or on the partition that is saying has no space)
  1. This should return whether there is space or not.
  2. If space is not available, expand partition or clear some pod logs.


Root Cause: /var partition becomes full and pods are impacted.


Resolution:

  • Run find /var/log/pods -type f -atime +10 -delete

This will delete the pod log folder and files and should clear up some space.