The issue involves a failure in starting the RKE2 server on a newly spun-up node within a 3-node HA production cluster. The failure occurred after the original node was removed due to cloud provider health checks.
Issue Description
The error message encountered in journalctl -u rke2-server:
"Waiting for other members to finish joining etcd cluster: etcdserver: unhealthy cluster"
Resolution
The solution involves removing the failed node from the etcd cluster using `etcdctl` commands. This allows the RKE2 service to start successfully on the new node.
ETCDCTL_CACERT="/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt"
ETCDCTL_CERT="/var/lib/rancher/rke2/server/tls/etcd/client.crt"
ETCDCTL_KEY="/var/lib/rancher/rke2/server/tls/etcd/client.key"
if [[ -z $ETCDCTL_BIN ]] ; then
etcdctl_bin=$(find /var/lib/rancher/rke2/agent/containerd/ -name "etcdctl" -type f -perm -u+x -print -quit)
[[ -z ${etcdctl_bin} ]] && warn "Skipping etcd operations as etcdctl not found" && return 0
ETCDCTL_BIN="${etcdctl_bin}"
fi
# List the members
${ETCDCTL_BIN} \
--cacert="${ETCDCTL_CACERT}" \
--cert="${ETCDCTL_CERT}" \
--key="${ETCDCTL_KEY}" \
member list
# Based on the list, figure out the member id for the node having the issue, and remove that member
${ETCDCTL_BIN} member remove <member-id>
Restart the rke2 service afterwards.