Configuring High Availability Add-On for Orchestrator Error

m_altourky · December 30, 2020, 11:23pm

hello everyone,

i am trying to implement orchestrator on a multi node high availability exact replica of the one mentioned in the link below:

and after installing the first three haa nodes following the instructions:

and after running the url:8443
i get this error:

when i navigate to the the linux server i run the rlcheck i got the following:

try to google it with no luck in fixing it.

thanks in advance to you all.

wasea · December 31, 2020, 12:11am

Hello @m_altourky,

I didn’t configured the orchestrator in HA mode, but looking at the screenshots I am thinking that is something related to the network part.

Did you opened the ports n order for the nodes to communicate properly?
I recommend to check this 2 things:

POrts as per :
Hardware and Software Requirements
(check with telnet or other tool that the requested ports are open between nodes)
webconfig file, to have the correct port.

I hope it helps.

Vasile.

m_altourky · December 31, 2020, 12:26am

thanks @wasea for your quick response, i appreciate it.
i did check some prots are open and some are closed, i did inform the network team about those ports before.

the webconfig already put the 8443 port for each node.

another thing please

i run the Orchestrator URL but did not get also any response except the output below from the url: /Account/Login?returnUrl=%2F

{“message”:“An error has occurred.”,“errorCode”:0,“resourceIds”:null}

what could be the issue?

JithinKP · December 31, 2020, 4:35am

@m_altourky - Can you please check the event viewer logs? Some useful log information will be there about the error.

m_altourky · December 31, 2020, 6:31am

@JithinKP - the logs refer that the issue is related to the error above
like this one

Blockquote
2020-12-31 05:13:58.6744 UiPath.IdentityServer.Web.Startup An error occurred during initialization of authentication methods
RedisConnectionExceptionIt was not possible to connect to the redis server(s). UnableToConnect on 10.15.61.219:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.0.593.37019 at StackExchange.Redis.ConnectionMultiplexer.ConnectImplAsync(Object configuration, TextWriter log)
at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.ConnectAsync(CancellationToken token)
at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefreshAsync(String key, Boolean getData, CancellationToken token)
at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAsync(String key, CancellationToken token)
at UiPath.IdentityServer.Application.Services.CustomDistributedCache.GetAsync[T](String key, Func`1 factory, CancellationToken token)
at UiPath.IdentityServer.Application.Services.ExternalIdentityProviderService.GetPartitionActiveIdpsAsync(Partition partition)
at UiPath.IdentityServer.Application.Services.ExternalIdentityProviderService.GetHostPartitionActiveExternalIdentityProvidersAsync()
at UiPath.IdentityServer.Web.Extensions.Configuration.ExternalAuthenticationMethodsExtension.AddAuthenticationMethodsAsync(AuthenticationBuilder builder, IServiceProvider serviceProvider, AppSettings appSettings)

JithinKP · December 31, 2020, 7:58am

@m_altourky - will you be able to try this resolution mentioned in SO?

m_altourky · December 31, 2020, 10:34pm

@JithinKP i did some search for that error

RedisConnectionException It was not possible to connect to the redis server(s). UnableToConnect on 10.15.61.219:6379 /Interactive,

as the Redis server (which installed by the UiPath HAA) is not using the 6379 port but 36379 this i think it is a bug in the UiPath HAA add-on.

thanks

JithinKP · January 1, 2021, 4:57am

@m_altourky - I don’t think it’s a bug. I did set up HAA twice and it was smooth (didn’t have to worry about pre-requisites though). We are missing something here. Only difference I can see is you are trying 2019 version and there may be some changes in 2020 version but that shouldn’t be a concern I guess. Let me see if I can try 2019 in Azure.

m_altourky · January 1, 2021, 4:39pm

@JithinKP
on Linux server i did the following:

rlcheck --continue-on-error

the result is :
Saving to file: /var/opt/redislabs/log/rlcheck.log
Running test: verify_bootstrap_status
PASS
Running test: verify_processes
Verifying local and master CCS…
PASS
Verifying CCS DNS…
Fatal error: Unable to connect to remote ‘/var/opt/redislabs/run/pdns.controlsocket’: Connection refused
FAILED - ccs_dns Failure: ccs_dns ping failed:
ERROR in test verify_processes: ccs_dns ping failed:
Running test: verify_dmcproxy
PASS
Running test: verify_port_range
PASS
Running test: verify_pidfiles
PASS
Running test: verify_capabilities
PASS
Running test: verify_existing_sockets
PASS
Running test: verify_host_settings
PASS
Running test: verify_tcp_connectivity
PASS
Summary:

FAILED TESTS:

verify_processes

although there was error regarding the tcp_connectivity but when i add the port 8001 to firewall it was gone.

running command redis-cli ping

result:

Could not connect to Redis at 127.0.0.1:6379: Connection refused
although the port in the config /opt/redislabs/config/ccs-redis.conf is 36379

in C:\Program Files (x86)\UiPath\Orchestrator\UiPath.Orchestrator.dll.config

<add key="LoadBalancer.UseRedis" value="true" />
<add key="LoadBalancer.Enabled" value="false" />
<add key="LoadBalancer.Redis.ConnectionString" value="10.15.61.219:36379,10.15.61.220:36379,10.15.61.221:36379,password=connectionpass@123" />

i tried every possible ports 10000,36379,6379
still getting the orchestrator error

{“message”:“An error has occurred.”,“errorCode”:0,“resourceIds”:null}

there was error on opining the /identity/ url
but when editing the C:\Program Files (x86)\UiPath\Orchestrator\Identity\appsettings.Production.json
and change the ports from 6379 to 10000 it worked on the /identity/ url and strart to ask for credentials. but not the orchastrator.

JithinKP · January 3, 2021, 6:32am

@m_altourky - Is HAA up now? Through Identity also you are not able to access orchestrator? Will you be able to check event viewer for that Orchestrator error?

Instead of us going through errors one by one, lets get UiPath Technical Support. It will save time and effort, hope you have an Enterprise Key for Orchestrator?

m_altourky · January 3, 2021, 5:14pm

@JithinKP HAA is up except the error it shows regarding PDNS.

no i can access the identity url but not the orchestrator.

there was errors before but related to HAA connections only nothing more.

believe me thats what i did since day one and till today i did not get even a ticket number or email notification from UiPath.

Nic_Surpatanu · January 3, 2021, 6:09pm

@m_altourky a few things to try:

UiPath.Orchestrator.dll.config files on all
the Orch nodes have the Redis config settings.
You have manually restarted the Orchestrator web site on all the nodes after you made the config changes (so the changes to config are picked up)

Thanks,
Nic

m_altourky · January 3, 2021, 10:09pm

@Nic_Surpatanu thank you for your time, the issue is am stuck with the first node of orchestrator so still the other node not installed yet as i am facing this error with the primary server.

Nic_Surpatanu · January 3, 2021, 11:59pm

@m_altourky Did you restart the website after you made config changes? It seems the config is not picked up. What is the error you have in event log after you restart and try to connect to Orch? Set the Orch diagnostics logging to Trace and take a look what is going on.

m_altourky · January 5, 2021, 9:00am

Dear @Nic_Surpatanu,
every time i made change to file inside the orchestrator directory i restart the IIS and make iisreset using command line as administrator sometimes.

regarding the orchestrator logs : IIs Event Log.zip (200.7 KB)

by the way i installed the secondary node of orchestrator and the logs is after that.
thanks to you all for help and support.

Nic_Surpatanu · January 5, 2021, 4:39pm

@rocsana can you please help @m_altourky?

rocsana · January 6, 2021, 7:35am

Hello @m_altourky

First of all, you need to be sure that you use the correct port for HAA. Based on our official documentation, HAA uses port 10000 by default & 10000-19999 is the port range for database traffic. I suggest to use 10000 port. Based on your last provided logs, port 36379 is used

After that, you need to be sure that BOTH Orchestrator & Identity config files are updated:

UiPath.Orchestrator.dll.config (C:\Program Files (x86)\UiPath\Orchestrator\UiPath.Orchestrator.dll.config)

<add key="LoadBalancer.UseRedis" value="true" />
<add key="LoadBalancer.Enabled" value="false" />
<add key="LoadBalancer.Redis.ConnectionString" value="redis:10000,password=pass" />

appsettings.Production.json (C:\Program Files (x86)\UiPath\Orchestrator\Identity\appsettings.Production.json)

“LoadBalancerSettings”: {
“RedisConnectionString”: “redis:10000,password=pass”,
“UseRedis”: “true”
}

Do you still have an error after you set the settings? If yes, please provide us the Event Viewer logs from the Orchestrator machine in order to debug them.

Thanks,
Rocsana

m_altourky · January 7, 2021, 5:47am

@Nic_Surpatanu, @JithinKP, @wasea, @rocsana thank you all for every thing i will reset all servers to default state and try to install All product again.

i did - as i mentioned early - use all ports in both setting files starting with 6379, 36379, 10000 with the last port number the only thing works is the /identity/ url.

i am now working on all servers again from scratch, so thank you for your time and support, i really appreciate it.

thanks one more time.
Mamoun.

Renato_Pinheiro_de_Souza · February 12, 2021, 7:46pm

Rocsana,

Do we need, in a HAA installation, put in both files the address of all redis/haa nodes? Right?

I saw the syntax of uipath.orchestrator.dll.config file in somewhere (lol) and was something like this:

<!--Load balancer-->
<add key="LoadBalancer.UseRedis" value="true" />
<add key="LoadBalancer.Enabled" value="false" />
<add key="LoadBalancer.Redis.ConnectionString" value="IP1:10000,IP2:10000,IP3:10000,password=password" />

But with appsettings.Production.json file, what do we need to use? Multiples lines?

“LoadBalancerSettings”: {
“RedisConnectionString”: “IP1:10000,password=password”,
“RedisConnectionString”: “IP2:10000,password=password”,
“RedisConnectionString”: “IP3:10000,password=password”,
“UseRedis”: “true”
}

For this settings would be better use a load balancer too? Something like AWS ELB.

Thanks in advance!

Renato Pinheiro

Renato_Pinheiro_de_Souza · February 12, 2021, 9:25pm

One more information! I’m testing with a AWS ELB and, till now, works like a charm.

Att.,
Renato

Topic		Replies	Views
Can't Connect Orchestrator to HAA Servers Orchestrator orchestrator , question	1	2274	March 2, 2021
スタンドアロン版Orchestratorの冗長構成でHAAへの自動再接続を有効化する方法ナレッジベース orchestrator	0	121	May 17, 2024
Recommendation on UiPath Orchestrator High Availability and DR - Redis and HA Addon Help orchestrator , question , infrastructure	5	3850	May 14, 2020
Need help on Orchestrator HIGH AVAILABILITY ADD-ON Orchestrator orchestrator , haa	4	1618	December 16, 2021
UiPath.IdentityServer.Web.Redis.RedisCache Orchestrator orchestrator , bug	3	859	February 28, 2023

Configuring High Availability Add-On for Orchestrator Error

Related topics