HAA - Multi Node with Azure VMs (Minimal Setup)

Hi,

I was trying to create this multi-node setup in Azure for sometime now. Thought I will consolidate and share it with you guys the working version I could deploy. This is a test setup and you can try it out in Azure to get an idea about multi-node setup. Let me know your thoughts and feedback’s.

Primary Requirement

  • An active Pay-As-You-Go Azure subscription

Architecture

System Requirements

License Requirements

Azure Resources to be created

Azure Network Security Groups and Ports to be enabled

Azure Resource Creation

Below setup is just based on my interest. You can try different network setups such as multiple resource groups/NSGs/Subnets/Rules etc.

  • Create a Resource Group - HAARG

  • Create Virtual Network and add the required no.of subnets based on your design.

  • Create a Private DNS Zone - This is to create the common Orchestrator URL DNS record. If we use the default Azure DNS zones it will have a pre-defined suffix which I wanted to avoid.

Zone name : novigoprivate.com

  • Link the VNET with Private DNS zone - Enable auto-registration if you want all the VMs to have a DNS record.

  • Create Network Security Groups

Make sure you are selecting the appropriate Subnets and NSGs when you create the virtual machines. We are going to have public ip only to the Stage VM. Rest all the VMs won’t have public ip and we will be accessing them from inside Stage VM using Private IPs.

To create F5 LTM, search for F5 BIG-IP Virtual Edition (BYOL) in market place and choose any of the 4 listed. We need only LTM and it will be available in all the 4 types listed.

Create all the VMs required together or one by one on the go.

Setup SQL Server

Connect to the SQL Server VM from Stage VM and install SQL Server 2019 Evaluation Edition. This is a generic SQL Server setup, you can find many tutorials in YouTube. Make sure to add Local Firewall Inbound Rule to enable traffic to 1433 port. Verify the connectivity using SSMS.

Orchestrator Node 1 Setup

We are setting up Single Node first and then upgrade to multi-node using Orchestrator LTS version (20.10.x) latest available.

$ssc = New-SelfSignedCertificate -FriendlyName “OrchestratorCertificate” -DnsName “orch-node1”, “orch-node1.novigoprivate.com
$store = New-Object System.Security.Cryptography.X509Certificates.X509Store( “Root”, “LocalMachine”)
$store.Open(“MaxAllowed”)
$store.Add($ssc)
$pass = ConvertTo-SecureString -String “Rpa@2021” -Force -AsPlainText
Export-PfxCertificate -Cert $ssc -FilePath “C:\Users\jithinkp\Documents\ssl\OrchestratorCertificate.pfx” -Password $pass

  • Make sure SQL Server is accessible from Orchestrator machine. Either try a telnet with 1433 or install SSMS in orchestrator machine and verify the connection.

  • Complete the Node 1 installation. Make sure you provide default hostname or DNS name because we are going to setup one orchestrator first and then change to multi-node. SSL created above is also applicable for only single node as there are no other SANs added.

I am going to use : orch-node1.novigoprivate.com, since we have the private zone available and VMs are auto-registered. SSL is also created with orch-node1.novigoprivate.com as SAN.

  • Verify the orchestrator is up and running.

  • Login to Host and Activate the Orchestrator Trial license.

Setting up HAA

We have 3 HAA Nodes to setup. One will be master and remaining two as secondary nodes. We are going to use RedHat VMs to setup Redis. Not a linux expert, so I m just following the pre-requisites provided here: Hardware and Software Requirements

By default Azure RedHat has firewalld enabled with public zone. Connect to Master Node using Putty and Private IP.

We need to enable the required ports in firewalld. This needs to be done in all the 3 nodes explicitly. I have added HTTP and HTTPS service extra because the installation uses APIs and internet to download HAA.

  • sudo su : (enable the root access)
  • firewall-cmd --get-active-zones
  • firewall-cmd --zone=public --add-port=3333-3339/tcp --permanent
  • firewall-cmd --zone=public --add-port=36379-36380/tcp --permanent
  • firewall-cmd --zone=public --add-port=8001/tcp --permanent
  • firewall-cmd --zone=public --add-port=8443/tcp --permanent
  • firewall-cmd --zone=public --add-port=8444/tcp --permanent
  • firewall-cmd --zone=public --add-port=9080-9081/tcp --permanent
  • firewall-cmd --zone=public --add-port=8070-8071/tcp --permanent
  • firewall-cmd --zone=public --add-port=9443/tcp --permanent
  • firewall-cmd --zone=public --add-port=10000-19999/tcp --permanent
  • firewall-cmd --zone=public --add-port=20000-29999/tcp --permanent
  • firewall-cmd --zone=public --add-port=53/tcp --permanent
  • firewall-cmd --zone=public --add-port=5353/tcp --permanent
  • firewall-cmd --zone=public --permanent --add-service=http
  • firewall-cmd --zone=public --permanent --add-service=https
  • firewall-cmd --reload
  • systemctl status firewalld

  • Set up Master Node : HAA-Node1

Follow the installation steps provided here : Installation

  • sudo swapoff -a

  • sudo sed -i.bak ‘/ swap / s/^(.*)$/#1/g’ /etc/fstab

  • wget --version (This is to check wget is already installed or not. By default it should be available and you can upgrade/install using “yum install wget” command if needed)

  • localectl (To check the local language. By default it’s “en_US.UTF-8”, if it’s different use the command "export LANG=“en_US.UTF-8"” to set it.

  • mkdir haa

  • cd haa

  • wget https://raw.githubusercontent.com/UiPath/Infrastructure/main/Setup/get-haa.sh

  • chmod +x get-haa.sh

  • ./get-haa.sh -u jithin.kp@novigo.com -p Rpa@2021 --verbose

    (./get-haa.sh -u email -p password --verbose)

(Since I m running these commands in root, I don’t need to specify sudo explicitly. verbose will give me the detailed log of installation.)

Verify if you are getting a successful message as below. Means your Master Node is up.

If you are facing any error, verify the log to check Redis Installation was successful or not first. Verbose will give the log on screen or look for install.log file.

If you are facing any issue at the end for creating the cluster or creating database, you can edit the get-has.sh script which is downloaded during this installation. It will be available in the folder you have created above. Edit the script using Vi or Vim editor and remove the Redis installation part. It’s another script “install.sh” inside the original script. Remove Redis part and run only the cluster creation and database part. It should give you a successful output as above.

  • Once master node is up, access the haa using “https://<master_ip>:8443” url. Login with the user id and password you have provided during installation. You should see something similar to below,

  • Set up Secondary Node 1 : HAA-Node2

  • Add the firewalld rules as mentioned before.

  • Follow the installation steps for secondary from docs.

  • sudo swapoff -a

  • sudo sed -i.bak ‘/ swap / s/^(.*)$/#1/g’ /etc/fstab

  • wget --version

  • localectl

  • mkdir haa

  • cd haa

  • wget https://raw.githubusercontent.com/UiPath/Infrastructure/main/Setup/get-haa.sh

  • chmod +x get-haa.sh

  • ./get-haa.sh -u jithin.kp@novigo.com -p Rpa@2021 -j 10.0.4.4 --verbose

    (./get-haa.sh -u email -p password -j IP_address_of_the_master_node --verbose)

  • Once the secondary node 1 is up, access the haa using “https://<second-node_ip:8443”. Login with the user id and password you have provided during installation. You should see something similar to below,

  • Set up Secondary Node 2 : HAA-Node3

Follow the same procedure of Secondary Node 1 to complete the installation of Secondary Node2.

  • Once the secondary node 2 is up, access the haa using “https://<second-node2_ip:8443”. Login with the user id and password you have provided during installation. You should see something similar to below,

Your HAA is up and running now.

Setting up NuGet Package Server - Minio

For multi-node setup, you need a common NuGet Package location for Orchestrators. We are going to use Minio as NuGet server as it’s free to setup.

  • Connect to Minio VM using putty and private ip.
  • sudo su

Minio uses port 9000 by default. Add firewalld rules to the port 9000 similar to before.

  • firewall-cmd --get-active-zones
  • firewall-cmd --zone=public --add-port=9000/tcp --permanent
  • firewall-cmd --reload
  • systemctl status firewalld

Download and install Minio Server.

  • wget https://dl.minio.io/server/minio/release/linux-amd64/minio
  • chmod +x minio
  • mv minio /usr/local/bin
  • adduser minio-user
  • mkdir /tmp/minio
  • chown -R minio-user:minio-user /tmp/minio/
    create a environment variable file which contains the user id and password to access minio.
  • vim /etc/default/minio - this is to create the file and add the below entries

MINIO_VOLUMES=“/tmp/minio/”
MINIO_OPTS=“–address :9000”
MINIO_ROOT_USER = jithinkp
MINIO_ROOT_PASSWORD = Rpa@2021

  • Save the file
  • cd /etc/systemd/system
  • vim minio.service - to create a minio service so that we can make the server to start running automatically. Add the below entries (modify accordingly).

[Unit]
Description=Minio
Documentation=https://docs.minio.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local/

User=minio-user
Group=minio-user

PermissionsStartOnly=true

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c “[ -n "${MINIO_VOLUMES}" ] || echo "Variable MINIO_VOLUMES not set in /etc/default/minio"”

ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

StandardOutput=journal
StandardError=inherit

#Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

#Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

#SIGTERM signal is used to stop Minio
KillSignal=SIGTERM

SendSIGKILL=no

SuccessExitStatus=0

[Install]
WantedBy=multi-user.target

  • Save the file.
  • systemctl daemon-reload
  • systemctl start minio.service
  • systemctl enable minio.service
  • systemctl status minio.service

You should get a successful message similar to below saying Minio Server is up and running.

  • Access the Minio Server using “http://minio-ip:9000” and login with the credentials you have set in Environment file.

  • Setting up Minio Client (MC)

Additionally, to work with Minio Server, you will need Minio Client (mc) to be installed. We will need to use MC to verify the server status, creating users, checking logs etc. It’s not mandatory for this test setup but I have added here if anyone needs it.

You can find more details here : MinIO Client — MinIO Object Storage for Linux

  • You can use the MC commands to perform various activities on your Minio Server now.

Setting up Orchestrator Node 1 with HAA and NuGet Server

Before starting the installation of Node 2 orchestrator, we are trying to make sure Node 1 works fine with HAA, NuGet and NLB setup.

  • Update HAA entries in UiPath.Orchestrator.dll.config file as mentioned in the docs - Installation

  • Similarly update HAA entries in AppSettings.Production.json file

  • Once changes are saved, restart the orchestrator from IIS and see if it’s working properly. If there is any connectivity issues, your orchestrator most probably won’t come up.

Setting up F5 load balancer

When you create an F5 Bring Your Own License (BYOL) setup, azure creates the F5 setup with single NIC. Means your F5 Configuration Utility URL and Virtual IP is going to be the same as your VM’s Private IP. You can get more details here : Microsoft Azure: Single NIC F5 BIG-IP Virtual Edition

  • Access the F5 console url : https://f5_private_ip:8443

  • Activate the license and make sure LTM feature is enabled.

Adding DNS record in Private Zone

Common URL for orchestrator multi-node setup will be tagged to the F5 virtual server ip. In our case, VIP and F5 IPs are same, so we need to add a DNS record to F5 ip with the URL name we need. I am going to use, rpa.novigoprivate.com as the DNS record and this will be my final multi-node URL.

  • In Azure, go to your private DNS zone created. Add a record set with the name of your choice.

Creating a Self-Signed SAN SSL Certificate

Since we don’t have a domain controller and machines are not connected to domains, we are going to create a Self-Signed certificate from Orchestrator Node 1 with all the required names in SAN. I am using the script provided in UiPath Docs. This will export a copy of the certificate for later use as well.

$ssc = New-SelfSignedCertificate -FriendlyName “CustomCertificate” -DnsName “rpa.novigoprivate.com”,“orch-node1.novigoprivate.com”,“orch-node2.novigoprivate.com”,“orch-node1”,“orch-node2”
$store = New-Object System.Security.Cryptography.X509Certificates.X509Store( “Root”, “LocalMachine”)
$store.Open(“MaxAllowed”)
$store.Add($ssc)
$pass = ConvertTo-SecureString -String “Rpa@2021” -Force -AsPlainText
Export-PfxCertificate -Cert $ssc -FilePath “C:\Users\jithinkp\Documents\ssl\CustomCertificate.pfx” -Password $pass

Changing Orchestrator/Identity URL

When setting up the multi-node, both orchestrators should have a common url and certificate needs to have the common url. We have the common url and certificate available now and Node 1 needs to be updated with this new details. Either you can update the config files one by one or you can use Platform Configuration Tool to run the script for update. Need to be careful with script as there is no rollback if it fails in mid-way.

Follow the steps provided here : Changing the Orchestrator / Identity Server URL

  • Update the new certificate thumbprint in AppSettings.Production.json file

  • Update the new URL in UiPath.Orchestrator.dll.config file

  • Update the new URL in AppSettings.Production.json file

  • Update the new URL in ClientRedirectUris table

  • Update the new URL in PostLogoutRedirectUri table

  • Restart the Orchestrator Node 1
  • Your Orchestrator Node 1 won’t come up at this point since the new URL is NLB url and we haven’t configured the NLB to divert the traffic.

Configuring F5 NLB to Self-Signed SAN SSL

  • Import the certificate to NLB

  • Add the imported certificates to both Client and Server Profiles

  • Add the certificate to ClientSSL and delete the Default entry certificate key chain.

  • Add the certificate to ServerSSL Profile

  • Update both Certificate and Key to the imported certificate

Configuring NLB to divert traffic to Orchestrator Node 1

  1. Create Node
  2. Create Pool
  3. Create Virtual Server

(We are using Round-Robin Algorithm)

You can find many tutorials in YouTube on below topics. I have used the one available here as reference : Microsoft Azure: Single NIC F5 BIG-IP Virtual Edition

  • Creating Node 1

  • Creating Pool

  • Creating Virtual Server

Verify Orchestrator Node 1 with Common URL

If you have previously stopped the Orchestrator, start it from IIS and try to access the new URL - https://rpa.novigoprivate.com

Note: If you are trying to access the URL from Stage-VM, make sure you have installed the Self-Signed SAN SSL certificate in Stage-VM, trusted root and personal locations.

It could take a while, but it should come up. Verify Event Viewer if you are finding any errors. Once the orchestrator is up, you should see a “secure” connection without any certificate errors. Both client side and server side is using the SAN SSL we have created, so there shouldn’t be any warnings.

Installing Orchestrator Secondary Node : Orch-Node2

We cannot directly install the secondary orchestrator with msi file. Secondary has to use same configurations of primary and same DB tables. UiPath has provided a script to generate the Parameter File which can be used for Secondary Node Installation.

  • You can find the script in Tools folder called Generate-ParametersFile

  • Script has an argument of file location. Run it with powershell by specifying a location to save the parameter file.

  • Parameter file will have all the attributes required for a secondary installation. Verify the details are proper or not.

  • Copy the parameter file and SAN SSL file to Orch-Node2.
  • Install the SAN SSL in Personal and Trusted Root locations.
  • Complete the pre-requisites for Orchestrator installation in Node 2
  • Install the Secondary Node using command line as given here : The Windows Installer

UiPathOrchestrator.msi SECONDARY_NODE=1 PARAMETERS_FILE=parameterFile.json /passive

It won’t take much time to complete the installation. With the above script, secondary will have an IIS binding of default host name. You will need to manually change the host name to “rap.novigoprivate.com” from “orch-node2”

  • Make sure the above value is changed to common url/both orchestrators are having same url in IIS as well as config files. DB is pointing to the same for both orchestrators.

  • You cannot verify the secondary orchestrator unless we setup NLB for secondary.

Configuring NLB for Multi-Node Orchestrators

Remove previously created Node/Pool/Virtual Servers. I couldn’t edit the Pool to add my secondary node, if you can find a way no need to delete them.

  1. Create two nodes for orchestrators
  2. Create a Pool and add the nodes to pool
  3. Create a virtual server and link it to the pool

(We are using Round-Robin Algorithm)

  • Access the orchestrator url multiple times / try with incognito.

  • Verify traffic in F5. Both Nodes should have traffic.

  • Alternatively, you can disable each node in F5 (or stop the orchestrator from IIS) and verify the connection is working to other node or not. Orchestrator should be accessible even if one of the node is down.

Connecting Robots and Executing a Test Process

As a final step, we need to connect the robot machine to orchestrator and verify it’s working fine or not.

  • Connect to the Robot-VM and install the SAN SSL certificate to both Personal and Trusted Root
  • Install Studio

  • Verify Orchestrator is accessible and Connect the Assistant to Orchestrator. If orchestrator is accessible and ports are opened, you should get connected without issues.

  • Create a test process in Studio and verify execution

  • Publish the project

  • Create a process and deploy to the assistant. Test attended execution.

  • Change the license to Unattended

  • Trigger the process and verify the execution

  • Lastly, check whether the package we published is getting stored in Minio or not.


That’s all…!!! :slight_smile:

In real-life scenario this kind of setup will never happen I guess but it can give you an idea about what we need or miss when setting up a multi-node. I didn’t have domain controller, CA certificates, proper F5 guidance etc, so suggestions and improvements are always welcome.

Hope this helps someone out there and save your time in digging…!! :slight_smile:

21 Likes

Hi @JithinKP

Thank you for the detailed steps and screenshots. It would be complete if Azure Key Vault is included to secure the appsettings.json files.

1 Like

@GreenTea - Thanks for the suggestion. Will explore Azure Key Vault features.!!

@JithinKP It’s Amazing! thank you for detailed instructions

1 Like

Hi JithinKP, thank you for this guide, are the HAA nodes you setup here needs license? is this the same as the paid Uipath HAA Multinode Plugin?

@Aldous_Deinla - Hi, by default when you install HAA, we get 30 days free license which is similar to the paid license. This can be replaced with actual license that is procured.

3 posts were split to a new topic: Azure Loadbalancer with High Availability Add-On