Overview
Purpose
Automatically escalate incidents to on-call engineers and sync acknowledgment/resolution actions.
What It Does
Automatically creates incidents in your paging system when Jira incidents are created
Syncs acknowledgment and resolution actions between Jira and your paging system
Routes incidents to the appropriate on-call team based on product ownership
Supported Platforms
PagerDuty
We support two-way sync with PagerDuty. When an incident is created or status changed in Phoenix Incidents, we create and update the status of the associated incident in PagerDuty. If an incident is marked as Acknowledged or Resolved in PagerDuty, the change is synced back to Jira.
Splunk On-Call
We support two-way sync with Splunk On-Call. When an incident is created or status changed in Phoenix Incidents, we create and update the status of the associated incident in Splunk On-Call. If an incident is marked as Acknowledged or Resolved in Splunk On-Call, the change is synced back to Jira.
OpsGenie
We support two-ways sync with OpsGenie--as long as you use a global Integration API Key. If you use a team-level integration key, only one-way sync is supported (and only one team will be able to be paged). When an incident is created or status changed in Phoenix Incidents, we create and update the status of the associated incident in OpsGenie. If an alert is marked as Acknowledged or Closed in OpsGenie, the change is synced back to Jira.
Note: These paging systems only support 3 statuses (Triggered, Acknowledged and Resolved), and that once an Incident is resolved, it can not be changed (unlike an Incident type in Jira). Opsgenie supports "unacknowledging" an alert, however Phoenix Incidents does not allow this, so this change will be ignored.
How to Configure
Select Paging Platform
Before you begin setting up your paging integration, you need to select which one you are using.
PagerDuty
You must key in two pieces of information, REST API Key & Bot Email
API KEY: Create an API KEY in PagerDuty and paste the key in REST API Key
Bot Email: Enter the email address that was used to create the API KEY in Bot email. The email must match the email that created the API KEY. We recommend using a bot/service account for this. Something that isn't tied to a specific user, so if that person leaves, the integration doesn't break.
Once the information is entered in, click Connect PagerDuty
Congratulations, you now have connected Phoenix Incidents to PagerDuty!
Splunk On-Call (previously VictorOps)
This is a 2 step process.
Step 1:
You must key in four piece of information
API ID: Create an API KEY in VictorOps and paste the API ID here
API KEY: Paste the API KEY you created above here
Organization Slug: This is a unique identifier found in your VictorOps URL (e.g.,
https://portal.victorops.com/client/{Org_ID}) and is used to identify and link your accountBot Username: This is the username of the user that configured the API KEY. The username must match the username that created the API KEY. We recommend using a bot/service account for this. Something that isn't tied to a specific user, so if that person leaves, the integration doesn't break.
Once the information is entered in, click Save Credentials
Step 2:
Follow the instructions and click Test Webhook
On successful connection, you will get the confirmation
Congratulations, you now have connected Phoenix Incidents to VictorOps!
OpsGenie
You must enter an Integration API Key and select your region (either US or EU).
OpsGenie Integrations Methods
In Opsgenie there are 3 ways to create integrations, the correct method is to use a Global-level API Integration.
Global API Integration.
These API Keys are created via Settings > Integrations > Add Integration. This is the method to correctly configure two-syn with Phoenix Incidents.
Team-level API Integration.
These API Keys are created via Teams > [select a team] > Integrations > Add Integration. This is a limited API key that will only support Jira -> OpsGenie one-way syncing and is not recommended.
Non-alert global API Key.
These API Keys are created via Settings > API Key management. These API Keys are not able to create alerts in OpsGenie and can not not be used.
How to create a API Integration in OpsGenie
Navigate to Settings > Integrations > Add Integration.
On the Integration selection screen, select "API".
Give the integration a name and make sure you leave the "Assignee team" as "None" so that it remains a global-level integration.
Press "Edit" on the newly created integration and make sure the following permissions are checked:
Read
Create/Update; and
Configuration access.
5. Copy the API Key.
6. Press the Save button on the modal and then Close button.
7. Enter the copied API Key into Phoenix Incidents, ensure you have selected the correct region and press the Connect Opsgenie button.
Congratulations, you now have connected Phoenix Incidents to OpsGenie!
Select Default Paging Target
Once you have connected your paging integration you should select the "default paging target". This is the Paging Service or Policy that we will page for incidents that are created when the impacted product has configured to a paging target.
It is strongly recommended to setup a default paging target just in case you ever have a product where the admin forgot to select the paging target in PagerDuty or Splunk On-Call.
A common choice is to choose your Devops or Platform team target, or choose the policy that represents your NOC if your company has one.
