How Can PreReq Pods in the EKS Automation Suite Be Troubleshot When They Enter a Pending State, Especially If PVC-related Pods Are Pending or Reporting Errors

News Knowledge Base
,
1

What steps can be taken to troubleshoot PreReq pods that remain in a Pending state—particularly PVC-related pods—when the following error occurs: 'Could not create volume 'pvc-…': could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 401'?

Issue:

What steps can be taken to troubleshoot PreReq pods that remain in a Pending state—particularly PVC-related pods—when the following error occurs: Could not create volume 'pvc-...': could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 401"

While running prereqs for Automation Suite on EKS, prereq pods are going into a pending state. Check the events of the pod using describe and see if it pending pvc creation. Describe the pvc and check if the events contain error below:

failed to provision volume with StorageClass "ebs-sc": rpc error: code = Internal desc = Could not create volume "pvc-xxx": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 401, request to EC2 IMDS failed. failed to get rate limit token, retry quota exceeded, 0 available, 5 requested

Resolution:

Issue could be due to lower value of HttpPutResponseHopLimit set in AWS, which could lead to credential fetching errors during volume provisioning, resulting in PVCs remaining in a pending state due to the inability to access EC2 instance metadata. To resolve this issue follow steps below

  1. Execute below query to get current value of HttpPutResponseHopLimit for EC2 instance where the PVC is provisioned:

    aws ec2 describe-instances --instance-ids <instance-id> --query "Reservations[].Instances[].MetadataOptions"

    You should see below output

    [

    {

    "State": "applied",

    "HttpTokens": "required",

    "HttpPutResponseHopLimit": 1,

    "HttpEndpoint": "enabled",

    "HttpProtocolIpv6": "disabled",

    "InstanceMetadataTags": "disabled"

    }

    ]

  2. Increase HttpPutResponseHopLimit to 2 by running the following command:

    aws ec2 modify-instance-metadata-options --instance-id <instance-id> --http-put-response-hop-limit 2

    You should see below output

    status:

    [

    {

    "State": "applied",

    "HttpTokens": "required",

    "HttpPutResponseHopLimit": 2,

    "HttpEndpoint": "enabled",

    "HttpProtocolIpv6": "disabled",

    "InstanceMetadataTags": "disabled"

    }

    ]

  3. Finally, check that PVCs are now bound and pods are functioning as expected:
    kubectl get pvc -A