HAA - “API authentication failed” - notification is getting piled up

[JithinKP]

Hi,
Can someone help us troubleshoot this issue? We have an active-passive setup and HAA is installed in a custom location than the root. On 3 Rhel8 nodes installation went smooth and all the 3 nodes are up and running but we could see a notification log piling up saying “API authentication failed (username: N/A)”. There are no errors in rlcheck or rladmin commands and not sure why this notification is coming up in every 3 sec or so. Has anyone faced this issue before?
@Nithin_P @codemonkee

All the nodes look healthy,

rlcheck shows pass on all the nodes,

rladmin status also shows OK,

[JithinKP]

Finally, issue is resolved after contacting UiPath Technical Support and Redis Team, below are the troubleshooting steps performed to resolve the issue.

1. run debug_mode on cnm_http
2. wait for 5 minutes.
3. Collect the support package. (You can verify using logs as well)
   
   
4. run debug_mode off cnm_http

After going through the support package, envoy_access.log of Node3 was showing 2 unknown IPs sending requests to Node3.

Checked with networking team regarding the 172.20.57.235 and 10.128.18.13 which identifies themselves as /officescan/cgi/isapiClient.dll. Seems like the Node3 IP was re-used one and previously it was allocated to some other activities. There were services from the above IPs still sending requesting to Node3 IP and it’s hitting HAA since the IP was re-used causing the authentication failure error.

Work-around : Uninstall HAA cluster, re-allocate a new un-used IP to Node3 and re-install the HAA. Removing only the Node3 from existing cluster is also possible but when you re-install HAA in Node3 and add it to the existing cluster, it will be added as “Node:4” which is done purposely by Redis team to track the cluster changes. To avoid confusion, we had to re-install the entire cluster with Node3 having a new IP.