Can't Connect Orchestrator to HAA Servers

Help Orchestrator
,
1

hello all,

in production environment everything was working fine and suddenly the Robots fails to connect to Orchestrator, when try to troubleshoot the issue find out that the orchestrator cannot connect to Redis server and the orchestrator provide the following error :

image
image2731×209 46.5 KB

after i restarted the HAA servers the following error appears ::

image
image1074×114 6.4 KB

the HAA shows the below errors:

image
image2698×602 133 KB

and the Web provide the following:

image
image1738×416 70.5 KB

the logs from the orchestrator like below:

*StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: SETEX n:2020.10.2.0_TenantLicense,c:1; UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321, mc: 1/1/0, mgr: 10 of 10 available, clientName: NICRPAUIORCH01, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=7,Free=32760,Min=4,Max=32767), v: 2.1.58.34321*
* ---> System.AggregateException: One or more errors occurred. (UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout)*
* ---> StackExchange.Redis.RedisConnectionException: UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321*
*   --- End of inner exception stack trace ---*
* ---> (Inner Exception #1) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

* ---> (Inner Exception #2) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

*   --- End of inner exception stack trace ---*
*   at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource`1 source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2777*
*--- End of stack trace from previous location where exception was thrown ---*
*   at Abp.Runtime.Caching.Redis.AbpRedisCache.SetAsync(String key, Object value, Nullable`1 slidingExpireTime, Nullable`1 absoluteExpireTime)*
*   at UiPath.Orchestrator.Redis.Caching.CacheExtensions.GetOrCreateAsync(ICache cache, String key, Func`2 factory)StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: SETEX n:2020.10.2.0_TenantLicense,c:1; UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321, mc: 1/1/0, mgr: 10 of 10 available, clientName: NICRPAUIORCH01, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=7,Free=32760,Min=4,Max=32767), v: 2.1.58.34321*
* ---> System.AggregateException: One or more errors occurred. (UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout)*
* ---> StackExchange.Redis.RedisConnectionException: UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321*
*   --- End of inner exception stack trace ---*
* ---> (Inner Exception #1) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

* ---> (Inner Exception #2) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

*   --- End of inner exception stack trace ---*
*   at void StackExchange.Redis.ConnectionMultiplexer.ThrowFailed<T>(TaskCompletionSource<T> source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2777*
*   at async Task Abp.Runtime.Caching.Redis.AbpRedisCache.SetAsync(string key, object value, Nullable<TimeSpan> slidingExpireTime, Nullable<TimeSpan> absoluteExpireTime)*
*   at async Task<object> UiPath.Orchestrator.Redis.Caching.CacheExtensions.GetOrCreateAsync(ICache cache, string key, Func<string, Task<object>> factory)*
*System.AggregateException: One or more errors occurred. (UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout)*
* ---> StackExchange.Redis.RedisConnectionException: UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321*
*   --- End of inner exception stack trace ---*
* ---> (Inner Exception #1) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

* ---> (Inner Exception #2) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

*StackExchange.Redis.RedisConnectionException: UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321*




*UiRedisCache.GetOrDefaultAsync failedStackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: GET n:2020.10.2.0_LicenseKeys,c:xxxxxxxx-49D6-4207-A07C-E007E6E51C17; UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321, mc: 1/1/0, mgr: 10 of 10 available, clientName: NICRPAUIORCH01, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=7,Free=32760,Min=4,Max=32767), v: 2.1.58.34321*
* ---> System.AggregateException: One or more errors occurred. (UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout)*
* ---> StackExchange.Redis.RedisConnectionException: UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321*
*   --- End of inner exception stack trace ---*
* ---> (Inner Exception #1) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

* ---> (Inner Exception #2) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

*   --- End of inner exception stack trace ---*
*   at void StackExchange.Redis.ConnectionMultiplexer.ThrowFailed<T>(TaskCompletionSource<T> source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2777*
*   at async Task<object> Abp.Runtime.Caching.Redis.AbpRedisCache.GetOrDefaultAsync(string key)*
*   at async Task<object> UiPath.Orchestrator.Redis.Caching.UiRedisCache.GetOrDefaultAsync(string key)*
*System.AggregateException: One or more errors occurred. (UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout) (It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout)*
* ---> StackExchange.Redis.RedisConnectionException: UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321*
*   --- End of inner exception stack trace ---*
* ---> (Inner Exception #1) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

* ---> (Inner Exception #2) StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. ConnectTimeout<---*

*StackExchange.Redis.RedisConnectionException: UnableToConnect (None, last-recv: 5) on 10.15.60.219:10000/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 7, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.1.58.34321*

thanks all for your support.

2

Hello @m_altourky!

It seems that you have trouble getting an answer to your question in the first 24 hours.
Let us give you a few hints and helpful links.

First, make sure you browsed through our Forum FAQ Beginner’s Guide. It will teach you what should be included in your topic.

You can check out some of our resources directly, see below:

  1. Always search first. It is the best way to quickly find your answer. Check out the image icon for that.
    Clicking the options button will let you set more specific topic search filters, i.e. only the ones with a solution.

  2. Topic that contains most common solutions with example project files can be found here.

  3. Read our official documentation where you can find a lot of information and instructions about each of our products:

  4. Watch the videos on our official YouTube channel for more visual tutorials.

  5. Meet us and our users on our Community Slack and ask your question there.

Hopefully this will let you easily find the solution/information you need. Once you have it, we would be happy if you could share your findings here and mark it as a solution. This will help other users find it in the future.

Thank you for helping us build our UiPath Community!

Cheers from your friendly
Forum_Staff