Setting Up PagerDuty As A Receiver In Alertmanager Within Rancher In Automation Suite

How to configure PagerDuty as a receiver in Alertmanager in a Rancher environment?

Issue Description: Automation Suite administrators may want to integrate PagerDuty with Alertmanager within Rancher to effectively manage and respond to alerts generated by their Automation Suite Kubernetes cluster.


Resolution:

Step 1: Obtain PagerDuty Integration Key

  1. Log in to your PagerDuty account.
  2. Navigate to the 'Automation' tab and click on the 'Event Orchestration' link
  1. Select the 'New Orchestration' button on the top-right section of the screen
  1. Fill in the 'name' (required) of the Orchestration Event; as well as the 'description' (optional) and 'permissions' fields (optional)
  1. In the next screen, go to the 'Integrations' tab and copy the 'Integration Key' (Routing Key) to the clipboard
image.png

Step 2: Configure Alertmanager in Rancher
  1. Log into the Rancher Dashboard UI (https://monitoring.automation-suite-fqdn)
  2. Navigate to the Prometheus/Alertmanager configuration, found under 'Monitoring' in your cluster’s dashboard. Click on the 'Routes and Receivers' submenu link
image.png
  1. Select the 'PagerDuty' option
image.png
  1. Paste in the 'Integration Key' obtained from the PagerDuty Orchestration Event created in the previous step
image.png**** The 'Enable send resolved alerts' option can result in notification fatigue (lots of flapping alerts)

Step 3: Check the Configuration
  1. Within PagerDuty, navigate to the 'Incidents Tab' and select the 'Alerts' section.