kubectl commands not working after Node Patching

News Knowledge Base
1

Kubeconfig no longer readable after non-reboot patching.

Issue:

#Performing any kubectl commands ends up with errors as below,

[root]$ kubectl get nodes

error: You must be logged in to the server (Unauthorized)

# rke2.yaml was configured in a custom location, seems new kubeconfig was generated during patching, that was corrupted.

[root]$ echo $KUBECONFIG

/home/asadmin/kube/auth/rke2.yaml

#More verbose logs:

[root]$ kubectl get nodes -v9

I0904 13:19:58.360465 2725949 loader.go:372] Config loaded from file: /home/asadmin/kube/auth/rke2.yaml

I0904 13:19:58.369334 2725949 round_trippers.go:435] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.22.12+rke2r1 (linux/amd64) kubernetes/b058e17" 'https://127.0.0.1:6443/api?timeout=32s'

I0904 13:19:58.371836 2725949 round_trippers.go:454] GET https://127.0.0.1:6443/api?timeout=32s 401 Unauthorized in 2 milliseconds

I0904 13:19:58.371854 2725949 round_trippers.go:460] Response Headers:

I0904 13:19:58.371859 2725949 round_trippers.go:463] Audit-Id: 987e61fd-b91b-4632-94cc-a3418db24a8f

I0904 13:19:58.371864 2725949 round_trippers.go:463] Cache-Control: no-cache, private

I0904 13:19:58.371868 2725949 round_trippers.go:463] Content-Type: application/json

I0904 13:19:58.371873 2725949 round_trippers.go:463] Content-Length: 129

I0904 13:19:58.371877 2725949 round_trippers.go:463] Date: Mon, 04 Sep 2023 13:19:58 GMT

I0904 13:19:58.372760 2725949 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

I0904 13:19:58.373650 2725949 cached_discovery.go:121] skipped caching discovery info due to Unauthorized

I0904 13:19:58.379162 2725949 helpers.go:217] server response object: [{

"metadata": {},

"status": "Failure",

"message": "Unauthorized",

"reason": "Unauthorized",

"code": 401

}]

F0904 13:19:58.379180 2725949 helpers.go:116] error: You must be logged in to the server (Unauthorized)

Resolution:

  • Replace the kubeconfig from original location to user home directory on the nodes with issue
  • Using a kubeconfig file from original location works fine, now test with:

kubectl --kubeconfig=/path/to/yaml get node