Configuring External Alerts via Alertmanager on AS 23.4 and above

News Knowledge Base
1

Can we configure External Alerts via Alertmanager on AS 23.4+?

This KB explains the steps to send alerts to external receivers. This guide demonstrates how to do so using email and Slack. For more details, refer to this documentation.

Notes:

  • Previously configured external alerts in 2022.10 or before will stop working because the monitoring stack was detached from what RKE2 provided.
  • If you configure alert forwarding with webhook and MS Teams, it will not work by default as mentioned in this StackOverflow page.

Receive Alerts via Email

As specified here, you can run the following command to forward alerts via SMTP:

./uipathctl config alerts add-email \

–name test \


–to “admin@example.com” \


–from “admin@example.com” \


–smtp server.mycompany.com \


–username admin \


–password somesecret \


–require-tls \


–ca-file <path_to_ca_file> \


–cert-file <path_to_cert_file> \


–key-file <path_to_key_file> \


–send-resolved

As of version 23.4.1, whether using --require-tls or not, the value of require-tls is set as true. If you need to disable require-tls, you need to manually edit alertmanager-config secret. For more details, refer to the following:

Set require-tls to False:

    • Backup alertmanager-config secret
kubectl -n cattle-monitoring-system get secret alertmanager-config -oyaml > alertmanager-config.yaml
    • Get the relevant config from alertmanager-config secret:
kubectl -n cattle-monitoring-system get secret alertmanager-config -ojson | jq -r '.data."alertmanager.yaml"' | base64 -d > alertmanager.yaml
    • In alertmanager.yaml’s email section (under email_configs), change the value of require_tls to false and save the file:
- name: test

email_configs:


    

  • send_resolved: true
    • 



to: <send_to email address>


from: 


hello: UiPath


smarthost: :


auth_username: 


auth_password: 


require_tls: false

    • To apply the change, run the following command
updated_alertmanager_config=$(cat alertmanager.yaml | base64 -w0)

kubectl -n cattle-monitoring-system patch secret alertmanager-config --type=‘json’ -p=‘[{“op” : “replace”, “path” : “/data/alertmanager.yaml”, “value” : "’“${updated_alertmanager_config}”‘"}]’

    • Run the following command to confirm that the change was successfully applied
kubectl -n cattle-monitoring-system get secret alertmanager-config -ojson | jq -r '.data."alertmanager.yaml"' | base64 -d

Confirmation that an alert email(s) was received:

image.png

Receive alerts via Slack

  1. There does not seem to be any way to configure alerts via uipathctl at this moment, so follow the steps below to manually forward alerts to your Slack organization:
    • Backup alertmanager-config secret
    • kubectl -n cattle-monitoring-system get secret alertmanager-config -oyaml > alertmanager-config.yaml
    • Get the relevant config from alertmanager-config secret
    • kubectl -n cattle-monitoring-system get secret alertmanager-config -ojson | jq -r '.data."alertmanager.yaml"' | base64 -d > alertmanager.yaml

  1. In alertmanager.yaml, add the following:
    • Under route.routes, add the entry like below. You can modify the value of severity to control what type of alerts to be forwarded. In the example below, it would forward alerts categorized as warning or critical 
      - receiver: "slack-notification"

      group_by:
      

        

      • ‘…’
        • 

      

      match_re:
      

      severity: warning|critical
      

      continue: true
      

      group_wait: 30s
      

      group_interval: 5m
      

      repeat_interval: 10d

    • Under receivers, add the following entry. Replace values wrapped by <> accordingly 
      - name: "slack-notification"

      slack_configs:
      

        

      • channel: “#”
        • 

      

      send_resolved: true
      

      api_url: “”



  1. Once the changes are done, save them.


 


  1. To apply the change, run the following command
  2. updated_alertmanager_config=$(cat alertmanager.yaml | base64 -w0) kubectl -n cattle-monitoring-system patch secret alertmanager-config --type='json' -p='[{"op" : "replace", "path" : "/data/alertmanager.yaml", "value" : "'"${updated_alertmanager_config}"'"}]'


 


  1. Run the following command to confirm that the change was successfully applied
  2. kubectl -n cattle-monitoring-system get secret alertmanager-config -ojson | jq -r '.data."alertmanager.yaml"' | base64 -d


 


Now, alerts should be sent to the Slack channel:


image.png