Linux Firewall Blocking Traffic To The Discovery Service Causing Orchestrator Failure

News Knowledge Base
1

Why am I encountering Redis failures causing my Orchestrator application to crash/fault?

Issue Synopsis:

Sometimes you may encounter intermittent/frequent Orchestrator failures such as while accessing the website, running jobs, connecting to Bots, etc. owing to Redis connection issues in a multi-node Orchestrator instance.

While there can be many scenario/environment-specific reasons for this, one of the very common reasons for this is when the Linux firewall blocks traffic to a port.

Some of the following exceptions may be seen in such a scenario:

  • MachineRuntimesCache GetOrDefaultAsync failedStackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: HGET
  • UiRedisCache.GetOrDefaultAsync failedStackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: PUBLISH
  • UiPath.IdentityServer.Web.Middleware.ExceptionHandlingMiddleware An error has occurred. RedisConnectionException*No connection is available to service this operation
  • StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). ConnectTimeout
  • StackExchange.Redis.RedisConnectionException: UnableToConnect on 10.152.109.60:10000/Interactive, Initializing/NotStarted
  • StackExchange.Redis.RedisConnectionException: SocketClosed
  • StackExchange.Redis.RedisConnectionException: SocketFailure

Issue Diagnosis and Resolution:

To diagnose this, perform a health check on the HAA nodes. Observe something like the following:

Also observe to see a similar TCP connectivity error/ warning within the HAA Console.

Also, observe that there is a connectivity issue for port 8001. This port is essentially used for traffic from the application to the Discovery Service. All the 3 Redis nodes would communicate to the discovery service using this port.

To resolve this, add a rule to allow traffic to port 8001. Refer to the "Configure the Firewall" section in this article for this: Redis Enterprise configuration.

Here is a list of all ports required for a seamless HAA connection:

Hence, to ensure that the HAA setup functions as expected at all times, ensure that none of the required ports are blocked.