Certificates validity check in Automation Suite with script

News Knowledge Base
1

How to check Certificates Validation with script in Automation Suite?

Description: Verify the validity of the necessary certificates using the script below.

Steps:

  1. Login as a root user to any server ( if multinode) and create a file
sudo su -
vi cert-validation-script.sh
  1. Copy the below content in the file
#!/bin/bash

export KUBECONFIG="/etc/rancher/rke2/rke2.yaml" \
&& export PATH="$PATH:/usr/local/bin:/var/lib/rancher/rke2/bin"

echo "####################Checking Server Certificate##############################"
echo "*******************************************************************************"
# Array of namespaces and secrets
declare -A SECRETS=( ["istio-system"]="istio-ingressgateway-certs" ["istio-system"]="istio-ca-cert"  ["uipath"]="istio-ca-cert" ["uipath"]="istio-ingressgateway-certs")

for NAMESPACE in "${!SECRETS[@]}"; do
  SECRET=${SECRETS[$NAMESPACE]}

  # Fetch the certificate data from the secret
  CERT_DATA=$(kubectl get secret $SECRET --namespace=$NAMESPACE -o jsonpath="{.data.tls\.crt}" 2>/dev/null)

  # If the certificate is not found, skip to the next iteration
  if [ -z "$CERT_DATA" ]; then
    echo "Certificate not found for secret $SECRET in namespace $NAMESPACE. Skipping..."
    continue
  fi

  # Decode the certificate data
  DECODED_CERT=$(echo $CERT_DATA | base64 --decode)

  # Write the decoded certificate data to a temporary file
  echo "$DECODED_CERT" > /tmp/cert.pem

  # Check the expiry of the certificate
  EXPIRY=$(openssl x509 -in /tmp/cert.pem -enddate -noout)

  echo "Secret $SECRET in namespace $NAMESPACE expires: $EXPIRY"
  # Get the expiry date of the certificate
  EXPIRY_DATE=$(openssl x509 -enddate -noout -in /tmp/cert.pem | cut -d= -f2)

  # Convert the expiry date to Unix timestamp
  EXPIRY_DATE_UNIX=$(date -d "$EXPIRY_DATE" +%s)

  # Get the current date as Unix timestamp
  CURRENT_DATE_UNIX=$(date +%s)

  # Check if the certificate has expired
  if [ $CURRENT_DATE_UNIX -gt $EXPIRY_DATE_UNIX ]; then
    echo "Certificate in secret $SECRET in namespace $NAMESPACE has expired."
  else
    echo "Certificate in secret $SECRET in namespace $NAMESPACE is valid."
  fi
done

echo "####################Checking Identity Certificate##############################"
echo "*******************************************************************************"
# Array of namespaces and secrets
declare -A SECRETS=( ["uipath"]="identity-token-signing-certificate")

for NAMESPACE in "${!SECRETS[@]}"; do
  SECRET=${SECRETS[$NAMESPACE]}

  # Fetch the certificate data from the secret
  CERT_DATA=$(kubectl get secret $SECRET --namespace=$NAMESPACE -o jsonpath="{.data.IdentityServer1\.pfx}"| base64 -d > /tmp/cert.pfx)

  # Fetch the password from the secret
  PASSWORD=$(kubectl -n $NAMESPACE get secret $SECRET -o jsonpath="{.data.password1}" | base64 -d)

  # Convert the PFX file to a PEM file

  openssl pkcs12 -in /tmp/cert.pfx -out /tmp/cert1.pem -passout pass:"${PASSWORD}" -passin pass:"${PASSWORD}"
 
  # Check the expiry of the certificate
  EXPIRY=$(openssl x509 -in /tmp/cert1.pem -enddate -noout)

  echo "Secret $SECRET in namespace $NAMESPACE expires: $EXPIRY"
  # Get the expiry date of the certificate
  EXPIRY_DATE=$(openssl x509 -enddate -noout -in /tmp/cert.pem | cut -d= -f2)

  # Convert the expiry date to Unix timestamp
  EXPIRY_DATE_UNIX=$(date -d "$EXPIRY_DATE" +%s)

  # Get the current date as Unix timestamp
  CURRENT_DATE_UNIX=$(date +%s)

  # Check if the certificate has expired
  if [ $CURRENT_DATE_UNIX -gt $EXPIRY_DATE_UNIX ]; then
    echo "Certificate in secret $SECRET in namespace $NAMESPACE has expired."
  else
    echo "Certificate in secret $SECRET in namespace $NAMESPACE is valid."
  fi
done
echo "####################Checking Webhook Certificate##############################"
echo "*******************************************************************************"
# Array of namespaces and secrets
declare -A SECRETS=( ["cert-manager"]="cert-manager-webhook-ca" ["cattle-system"]="cattle-webhook-tls")

for NAMESPACE in "${!SECRETS[@]}"; do
  SECRET=${SECRETS[$NAMESPACE]}

  # Fetch the certificate data from the secret
  CERT_DATA=$(kubectl get secret $SECRET --namespace=$NAMESPACE -o jsonpath="{.data.tls\.crt}" 2>/dev/null)

  # If the certificate is not found, skip to the next iteration
  if [ -z "$CERT_DATA" ]; then
    echo "Certificate not found for secret $SECRET in namespace $NAMESPACE. Skipping..."
    continue
  fi

  # Decode the certificate data
  DECODED_CERT=$(echo $CERT_DATA | base64 --decode)

  # Write the decoded certificate data to a temporary file
  echo "$DECODED_CERT" > /tmp/cert2.pem

  # Check the expiry of the certificate
  EXPIRY=$(openssl x509 -in /tmp/cert2.pem -enddate -noout)

  echo "Secret $SECRET in namespace $NAMESPACE expires: $EXPIRY"
  # Get the expiry date of the certificate
  EXPIRY_DATE=$(openssl x509 -enddate -noout -in /tmp/cert.pem | cut -d= -f2)

  # Convert the expiry date to Unix timestamp
  EXPIRY_DATE_UNIX=$(date -d "$EXPIRY_DATE" +%s)

  # Get the current date as Unix timestamp
  CURRENT_DATE_UNIX=$(date +%s)

  # Check if the certificate has expired
  if [ $CURRENT_DATE_UNIX -gt $EXPIRY_DATE_UNIX ]; then
    echo "Certificate in secret $SECRET in namespace $NAMESPACE has expired."
  else
    echo "Certificate in secret $SECRET in namespace $NAMESPACE is valid."
  fi
done

echo "####################Checking Rke2 Certificate##############################"
echo "*******************************************************************************"
# Directory containing the .crt files
DIR="/var/lib/rancher/rke2/server/tls"

# Get the current date as Unix timestamp
CURRENT_DATE_UNIX=$(date +%s)

# Loop over all .crt files in the directory
for CERT in $DIR/*.crt; do
  # Get the expiry date of the certificate
  EXPIRY_DATE=$(openssl x509 -enddate -noout -in "$CERT" | cut -d= -f2)

  # Print the expiry date
  echo "Expiry date of $CERT: $EXPIRY_DATE"

  # Convert the expiry date to Unix timestamp
  EXPIRY_DATE_UNIX=$(date -d "$EXPIRY_DATE" +%s)

  # Check if the certificate has expired
  if [ $CURRENT_DATE_UNIX -gt $EXPIRY_DATE_UNIX ]; then
    echo "Certificate $CERT has expired."
  else
    echo "Certificate $CERT is valid."
  fi
done
rm -rf /tmp/cert.pem /tmp/cert1.pem /tmp/cert2.pem /tmp/cert.pfx

  1. Save and exit the file ( "esc" ":" "wq!" hit "enter")
  2. Grant the execution permissions to the file and run the script
chmod +x cert-validation-script.sh
./cert-validation-script.sh

​​​​

The script will verify the below certificates:

  1. Server
  2. Identity
  3. Webhook
  4. Rke2 Rancher

To update the certificates, do the following if any of the above have expired.

  1. Server: Follow the Documentation
  • https://docs.uipath.com/automation-suite/automation-suite/2023.10/installation-guide/managing-the-certificates#updating-the-server-certificate

(change the version as per AS version)

  1. Identity:
  1. Navigate to the installation folder (likely found here: /opt/UiPathAutomationSuite/)
  2. Optional -It is easier to make a new folder in the installer directory:
mkdir certs
While in the installer directory (not the new certs folder) run the command - update the online installer to your version:
sudo ./configureUiPathAS.sh tls-cert get --outpath /opt/UiPathAutomationSuite//certs

Now in the /certs folder, see tls.crt and tls.key. Run the following command:

openssl pkcs12 -export -out identity.pfx -inkey tls.key -in tls.crt

There will be a prompt to set a password, make note of this password.
(You'll then have the new identity.pfx file in the /certs directory)

  • Add the New Identity Certificate to Automation Suite

To add the new certificate, go back to the installer directory and run the following command (you’ll need to update the script to your installer version again and update the password with the one just created):

sudo ./configureUiPathAS.sh identity token-cert update --cert-file-path /opt/UiPathAutomationSuite//certs --password
  • Rotate the New Identity Certificate for the Expired Version

Next rotate the expired cert for the new one added. Run the command:

sudo ./configureUiPathAS.sh identity token-cert rotate

To validate re-run the above script

  1. Webhook:
  • Rotation of Expired webhook certificates, run the below commands
    • Enable kubectl
sudo su -
export KUBECONFIG="/etc/rancher/rke2/rke2.yaml" \
&& export PATH="$PATH:/usr/local/bin:/var/lib/rancher/rke2/bin"

Rotate expired Certificate,

kubectl delete secret -n cattle-system cattle-webhook-tls
kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io --ignore-not-found=true rancher.cattle.io
kubectl delete pod -n cattle-system -l app=rancher-webhook

Ref: https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/expired-webhook-certificate-rotation

  1. Rke2 Certificate:

Default expiry period of RKE k8s certificate is 1 year and server-ca.crt is 10 year. If k8s certificates are expired then cluster won't be accessible

If client-rke2-controller.crt is expired and not valid follow below steps

  • Run below commands on first master node. To identify first node, run below command ( for multinode)
cat /etc/rancher/rke2/config.yaml |grep -i ^server

Output of above command should be empty for first server node.

Run below commands on first master node,

  1. Run systemctl stop rke2-server.service to stop rke2 server
  2. Run rke2-killall.sh command to cleanup any remaining process from rke2
  3. Delete /var/lib/rancher/rke2/server/tls/dynamic-cert.json file. (Only on server nodes)
  4. Start rke2 server using systemctl start rke2-server.service
  5. Delete rke2-serving secret from kube-system namespace using kubectl delete secret -n kube-system rke2-serving command. (Note: kubectl commands may not be accessible in case of multi-node deployment until steps from 1st to 4th are executed, on required number of server nodes - for etcd quorum requirement. Delete rke2-serving secret as soon as rke2 server boot up.)

If cluster is having more than one server node then above systemctl start rke2-server.service command may not execute till completion as etcd will not be able to complete leader election. You can start executing same steps, from 1st to 4th, on other server nodes.

Once quorum for etcd will satisfy, rke2 server will be able to start rest of the control plane pods and you should see kubectl get nodes command passing.

Once server nodes are ready, you can perform below steps on agent nodes to recreate certificates.

  1. Run systemctl stop rke2-agent.service
  2. Run rke2-killall.sh
  3. Run systemctl start rke2-agent.service

If server-ca.crt is expired and not valid then follow below

Run below commands on first master node. To identify first node, run below command

cat /etc/rancher/rke2/config.yaml |grep -i ^server

Output of above command should be empty for first server node.

Run below commands on first master node

  1. Run systemctl stop rke2-agent.service
  2. Run rke2-killall.sh
  3. Run systemctl start rke2-agent.service