Debugging Insights Certificate Issues

Challenges encountered with a certificate update for the Insights application.


Sometimes challenges with the Insights application are encountered post an Insights/Looker certificate change. Review the following steps to further investigate and resolve the issue.

How Certificates Work:

Many times, there are challenges in managing HTTPS certificates utilized for TLS encryption between client and server.The issue often arises due to a lack of clarity regarding the certificate's purpose and its associated requirements. Here's a very brief primer on the key details to know:

  1. The purpose of certificates is to make the internet safer (among other things).
  2. Anytime a browser connects to a website using HTTPS, a certificate is involved. A secure site looks like:
image.png
  1. A certificate is a document that can be used to establish if a website is trustworthy (It's really just a long random looking string). How that trust is established is done using complex mathematical operations. But for our purposes, we need to know the following:
    1. Root Certificates are certificates that we trust directly. For these certificates, there exists an actual file on our computer that contains the certificate. Because the file exists on our computer in the right location, we trust it.
    2. Certificates can be chained together. So if Certificate A is present, which is trusted and it says to trust certificate B, then trust Certificate B. The correct term for this would be that Certificate A issued Certificate B. This can be extended further.
    3. When we connect to a site, it presents a certificate to the browser, if that matches one of the Root Certificates or if its issued by one of our Root Certificates, it can be trusted. If the site is trusted, then it is marked as secure by the browser.​​
    4. Behind the scenes, every certificate has a private key. This is used to issue certificates and it should never be shared. ( Share a certificate without sharing the private key)
  2. Certificates also have metadata. In the context of HTTPS the most important attribute is called the 'Subject Alternative Name' (SAN). This is the name of the server that is being connected to (usually the FQDN). Some admins might simple refer to this as the DNS name. For any URL, determine what the value of the SAN should be. Here are a few examples:
    1. https://google.com -> The SAN should be google.com
    2. https://uipath.com/docs -> The SAN should be uipath.com
    3. https://insights.mycompany.com/home -> The SAN should be insights.mycompany.com. The domain would be mycompany.com.
    4. https://insightslinux.mycompany.com/login -> The SAN should be insightslinux.mycompany.com. The domain would be mycompany.com.
  3. If the SAN does not match the URL being used, a browser will mark the site insecure (even if it trusts the certificate).
  4. A certificate can have multiple SAN entries. The SAN entry is always a valid DNS name.
  5. A few other things:
    1. There are what is called Public Root Certificates. These come from well well-known organizations (DigiCert, AWS, CloudFlare, etc). They not only issue certificates but they ensure that they only issue certificates to websites that own the domain. For example, no one but the company UiPath can request a certificate with the SAN attribute uipath.com from a public Root Certificate company because we own the domain.
    2. Internally within a company's intranet, a company can have its own Root Certificates. These are Private Root Certificates but are like public root certificates in the context of the company's intranet. Companies usually have an internal root certificate provider that is used for internal sites. This allows them to issue certificates for internal sites without having to buy certificates from a public certificate authority. For example, they might issue a certificate for a site called: rpa.mycompany.internal.
    3. For companies with internal root certificates, usually automatically install these certificates onto the machines connected to their intranet.
  6. This means that for a site to be secure the following conditions must be met:
    1. The SAN attribute matches the server name in the URL.
    2. The certificate presented by the website needs to be a Root Certificate or issued by a Root Certificate.
    3. This also means that UiPath cannot provide these certificates for end users. For us to issue a certificate for an on prem environment, either, we would have to own the domain or we would need to be given access to the company's internal Root Certificate's private key, which is something that should never be shared.
Create self-signed certs. These are new Root Certificates. However, these certificates would have to be installed on every user's machine manually and tend to lead to more complications. For example, if the certificate is not installed on a user's machine, they can usually bypass the certificate warning, but many other things would break.
  1. With the above in mind, these are the important things to know:
    1. Almost all companies have an internal root CA (I have never encountered a company that does not).
    2. Almost all companies have a well defined well-defined process for requesting a certificate issued by an internal root certificate. The biggest challenges are:
      1. Being aware of this process
      2. Understanding the SAN attribute and requesting the server name correctly.
    3. The same process used to issue a certificate for a Windows machine can be used for a certificate that will be used on a Linux machine.


Troubleshooting/Diagnosing:

  1. Note that Insights has two web servers. The web server is hosted on the Windows machine and the web server is hosted on the Linux machine. The Windows machine is the frontend service and the Linux machine is the backend service. However, a browser connected to Insights will connect to both services.
    1. See: Insights Architecture
    2. This means that both services need to have a valid certificate for HTTPS communication (It is possible to use one certificate, but that certificate has to have valid SAN names for both the Windows and Linux servers)
    3. Additionally, for both services the following applies: Using a Certificate for the HTTPS Protocol
  2. This article will address the Windows portion first
    1. For Windows, a few things to note:
      1. The URL hostname is defined at install time when the MSI asks for the Insights server URL. See: MSI Installer
      2. The settings provided at this stage are used to configure the IIS binding for the Insights site. IIS bindings are described here: IIS Binding Documentation.
    2. To see if there is a problem with the certificate, simply go to the Insights URL. If there is a problem with the certificate, the browser will display an error. In most cases, the browser will display a helpful error message.
      1. See Certificate Error Article (This article is used to diagnose robot errors, but the debugging steps are the same. Just use the Insights URL instead of the Orchestrator URL).
      2. Capture screenshots (and make sure to include the URL)
    3. If the issue is that the certificate is expired, or that the certificate needs to be updated follow: Updating the Portal Certificate in IIS.
If issues are encountered during the update, capture screenshots.
  1. If the issue is still not resolved for the Windows component, please raise a ticket. Include the following:
    1. Screenshots that were captured in the previous steps
    2. If the certificate shows insecure, try exporting the certificate. This can be done by clicking the 'Not secure' icon in the browser and viewing the certificate. In the certificate viewer, there should be an option to export. This is just the public key and not the private key, so this is safe
    3. Screenshot of the IIS binding settings from step 4 in the certificate change article
    4. Any other information on the steps that were taken.
  2. For the Linux portion:
    1. To see if there is a problem with the certificate, simply go to the Insights Looker URL.
      1. See Certificate Error Article (This article is used to diagnose robot errors, but the debugging steps are the same. Just use the Insights URL instead of the Orchestrator URL).
      2. Capture screenshots (and make sure to include the URL)
    2. For updates verify if these steps are followed: Looker Insights Certificate Update.
      1. If these steps did not work, take a screenshot of the pre-installation tool and any errors encountered
      2. If there were any steps that were performed outside of the above, elaborate on the same
      3. Consider leveraging the certificate change script as an alternative way to change your certificate. See the section 'Certificate Change Script'
    3. If none of the above steps resolves, provide the following:
      1. Share the following logs with us for further analysis: How To Collect All Looker Logs For Insights?
      2. Screenshots that were captured in the previous steps
      3. If the certificate shows insecure, try exporting the certificate. This can be done by click the 'Not secure' icon in the browser and viewing the certificate. In the certificate viewer there should be an option to export. This is just the public key and not the private key, so this is safe
      4. Any other information on the steps that were taken.

Certificate Change Script:
  1. The script takes as input the linux server name and a pfx file.
  2. The server name is required so that the certificate can be validated.
  3. i.e. sudo certificateUpdate.sh -c cert.pfx -s
#!/bin/bash

usage() {
    echo "Script for updating the insights cert.pfx file"
    echo "Usage: $0 -c  -s "
    echo "  -c    Specify the PFX file."
    echo "  -s    Specify the server name of the linux server"
    echo ""
    echo "NOTE: The certificate should contain a SAN attribue for the linux server name."
    echo "      Additionally it should be a FQDN name"
    exit 1
}

# check that valid arguments are given.
function check_args() {
    if [[ -z "$PFX_FILE" || -z "$FQDN" ]]; then
    echo "FATAL: Both PFX_FILE (-c) and server name (-s) must be provided."
    usage
    fi
}

# Check if a valid pfx file is given as input
function check_pfx_exists() {

    echo "Checking that the PFX file exists at: $PFX_FILE"
    if [[ ! -e "$PFX_FILE" ]]; then
        echo "FATAL: The provided PFX file: $PFX_FILE does not exist."
        exit 1
    fi
    echo "  PFX file exists"
    echo ""

}

# Get the password for the PFX
function get_pfx_password() {

    echo "Checking for PFX password"
    echo "  Executing: openssl pkcs12 -info -in "$PFX_FILE" -noout -passin pass: &>/dev/null"
    # Try to read PFX info without a password
    openssl pkcs12 -info -in "$PFX_FILE" -noout -passin pass:"" &>/dev/null

    # Get the password and then verify the password
    if [ $? -ne 0 ]; then
        echo "Please enter the password for the PFX file:"
        read -s PFX_PASSWORD

        echo ""
        echo "  Checking PFX password: openssl pkcs12 -info -in "$PFX_FILE" -noout -passin pass: **** 1>/dev/null"
        openssl pkcs12 -info -in "$PFX_FILE" -noout -passin pass:$PFX_PASSWORD &>/dev/null
        if [ $? -ne 0 ]; then
            echo "FATAL: Failed to access PFX with given password"
            echo "FATAL: Check password or try running above command manually"
            exit 1
        fi
    else 
        echo "  No password needed"
    fi
    
    echo ""
}

# Check that the pfx has a valid san attribue
check_san_from_pfx() {

    echo "Checking the SAN attribue"
    echo "  Executing: openssl pkcs12 -in "$PFX_FILE" -passin pass:**** -nokeys -clcerts -out tmp.pem"
    # Extract the certificate from the PFX file
    openssl pkcs12 -in "$PFX_FILE" -passin pass:$PFX_PASSWORD -nokeys -clcerts -out /tmp/tmp.pem

    if [ $? -ne 0 ]; then
        echo "FATAL: Failed to create pem formated file"
        exit 1
    fi

    # Extract the SAN information
    echo "  Extracting the SAN attribue: openssl x509 -in /tmp/tmp.pem -noout -text | grep -A 1 "Subject Alternative Name:" | tail -n1"
    local san=$(openssl x509 -in /tmp/tmp.pem -noout -text | grep -A 1 "Subject Alternative Name:" | tail -n1)

    if [ $? -ne 0 ]; then
        echo "FATAL: Failed to read SAN info"
        echo "Cleaning up /tmp/tmp.pem"
        rm /tmp/tmp.pem
        exit 1
    fi

    # Clean up the temporary certificate
    echo "  Removing temp cert: /tmp/tmp.pem"
    rm -f /tmp/tmp.pem

    
    # Print the SAN (or you can process it further if needed)
    if [[ $san =~ $FQDN ]]; then
        echo "  The certificate has a valid san attribute for $FQDN"
    else
        echo "FATAL: The certificate does not have a valid san attribute for: $FQDN"
        echo "FATAL: The san attributes are: $san"
        exit 1
    fi

    echo ""
 
}


# Convert the PFX to a PFX with no password
function convert_pfx() {
    
    echo "Converting the PFX for installation"

    # Run this inside the container so we do not have FIPS issues
    echo "  Extracting cert in pem format."
    echo "  Executing: openssl pkcs12 -in $PFX_FILE -out $INSIGHTS_DIR/tmp.pem -nodes -passin pass:***"
    openssl pkcs12 -in $PFX_FILE -out $INSIGHTS_DIR/tmp.pem -nodes -passin pass:$PFX_PASSWORD

    if [ $? -ne 0 ]; then
        echo "FATAL: Extration command failed!"
        echo "FATAL: Checking permissons on $PWD"
        exit 1
    fi

    echo "  Executing: sudo chmod 744 $INSIGHTS_DIR/tmp.pem"
        if [ $? -ne 0 ]; then
        echo "FATAL: Failed to set perms on $INSIGHTS_DIR/tmp.pem"
        echo "FATAL: Checking permissons on $PWD"
        exit 1
    fi

    echo "  Creating new cert.pfx at $INSIGHTS_DIR/cert.pfx"
    echo "    Checking if $INSIGHTS_DIR/cert.pfx exists"
    if [[ -e "$INSIGHTS_DIR/cert.pfx" ]]; then
        randomN=$RANDOM
        echo "    The file $INSIGHTS_DIR/cert.pfx exists moving to $INSIGHTS_DIR/cert.pfx_$randomN"
        echo "    Executing: sudo mv $INSIGHTS_DIR/cert.pfx $INSIGHTS_DIR/cert.pfx_$randomN"
        sudo mv $INSIGHTS_DIR/cert.pfx $INSIGHTS_DIR/cert.pfx_$randomN
        if [ $? -ne 0 ]; then
            echo "FATAL: Could not move file"
            echo "Cleaning up /tmp/tmp.pem"
            rm "/tmp/tmp.pem"
            exit 1
        fi
    fi

    echo "  Creating new cert.pfx"
    echo "    Executing: sudo docker exec -it looker-container openssl pkcs12 -export -in /app/.deploy/tmp.pem -out /app/.deploy/cert.pfx -passout pass:"
    sudo docker exec -it looker-container openssl pkcs12 -export -in /app/.deploy/tmp.pem -out /app/.deploy/cert.pfx -passout pass:

    if [[ $? -ne 0 ]]; then
        echo "Fatal: Failed to create $INSIGHTS_DIR/cert.pfx"
        echo "Cleaning up $INSIGHTS_DIR/tmp.pem"
        rm "$INSIGHTS_DIR/tmp.pem"
        exit 1
    fi 
    
    echo "    Executing: sudo chmod 744 $INSIGHTS_DIR/cert.pfx"
    sudo chmod 744 $INSIGHTS_DIR/cert.pfx
    
    if [[ $? -ne 0 ]]; then 
       echo "Fatal: Failed to set perms on $INSIGHTS_DIR/cert.pfx" 
       exit 1
    fi
    
    
    echo "  Successfully created $INSIGHTS_DIR/cert.pfx"
    echo ""


}


# Check if a specific docker container is running
function check_docker_container() {

    echo "Checking that the docker container is running: sudo  docker ps -q -f name=looker-container"
    if [[ -z $(sudo docker ps -q -f name=looker-container) ]]; then
        echo "FATAL: Docker container $CONTAINER_NAME is not running."
        echo "FATAL: Try: sudo systemctl restart docker"
        exit 1
    fi

    echo "  Docker container is running"
    echo ""
}

# Fine the location of insights
function set_insights_dir() {
    echo "Setting insights working directory"
    echo "  Executing: sudo docker inspect looker-container | jq -r '.[0].HostConfig.Binds' | grep 'deploy' | awk -F':/app/.deploy' '{print $1}' | awk -F'\"' '{print $2}'"
    INSIGHTS_DIR=$(sudo docker inspect looker-container | jq -r '.[0].HostConfig.Binds' | grep 'deploy' | awk -F':/app/.deploy' '{print $1}' | awk -F'"' '{print $2}')
    
    if [ $? -ne 0 ]; then
        echo "FATAL: Could not set the insights working directory"
        echo "FATAL: Try running command manually to see the error"
        exit 1
    fi

    echo "  Verifying that $INSIGHTS_DIR exists"
    if [[ ! -e $INSIGHTS_DIR ]]; then
        echo "FATAL: Insights directory is missing!"
        echo "FATAL: Open a ticket with UiPath to repair this"
        exit 1
    fi 

    echo ""
}

# Run the certificate update
function run_certificate_update() {
    echo "Running certificate update"
    echo "  Executing: sudo docker exec -it looker-container su looker -m -c '</app/looker-init-job/install-certificate.sh bash'"

   sudo docker exec -it looker-container su looker -m -c '&2
            usage
            ;;
        :)
            echo "Option -$OPTARG requires an argument." >&2
            usage
            ;;
    esac
done


# Main execution
check_args
echo "Starting Certificate Rotation Process"
echo ""
check_pfx_exists
get_pfx_password
check_san_from_pfx
check_docker_container
set_insights_dir
convert_pfx
run_certificate_update
echo "Complated Certificate Rotation Process, certificate should be updated"