Create a Grafana Alert
In this document, we'll walk through the steps to create an alert with Grafana.
Use case: When the Grafana alert fires, we would like to run an unSkript RunBook to remediate the situation.
The high level steps for creating a Grafana alert:
These are the parameters that will cause the alert to fire.
Creating a Grafana Contact point into unSkript
The contact information - where should the alert fire?
How often will the alert notify unSkript?
Sent test data to unSkript.
Each of these high level steps are described in greater detail in this page:
Creating Grafana Rule
First we must create the alert at Grafana. If you already have the alert you wish to use, you can skip this section.
On this page, we will create a new rule.
Give your Rule a name, and rule type ('Grafana managed alert')
Create alert queries.
This query is reading a Postgres database for the count of Github stars for Awesome-Cloudops-Automation (click the link, and give us a star!)
The second query is going to take the results from the first query, and if the last value is over 200, the alert will fire. (There are more than 200 stars, so this alert will begin firing immediately.)
Define the alert conditions. In the screenshot below, this will check every 1 minute, and begin firing the error after 5 minutes.
Note: This frequency is much too fast this alert. This frequency will be useful while provisioning the alert. We will update the conditions once the alert is set.
Step 4 of the process is to add details. Add any details that you wish, and the save the alert.
Create the Contact Point
Next, we need to connect the alert to unSkript.
If you have not yet created a Grafana webhook, Create a Webhook with one of your Grafana Credentials.
In the Grafana dashboard, we need to create a contact point. This can be found in the alarms menu in the left navigation:
Create a new Contact point.
Type: Webhook
url: The webhook url provided by unSkript
HTTP method: POST
Username (from unskript webhook details)
Password (from unSkriupt webhook details)
Save your contact point.
Notification Policy
Ok, so we've created an alert, and a connection to unSkript. Now let's build a notification policy.
Grafana is set with a default notification policy, or you can set an individual policy for specific alerts to be sent to unSkript.
Let's create a specific routing for this alert:
Matching rules: Select the alert. Set label to "alertname" and the Value to the name of your alert (in the screenshot below, the alert name is "Docs_rule.")
Contact Point: Select the Contact point you created with unSkript webhook credentials.
Override general settings. These settings will force the alarm to fire every 2 minutes. That is very frequent, but useful when setting up your alerts. Choose settings that make sense for your alert.
Test the alarm
Send a test alarm message. In the Connection Points menu, select the connection pointing to unSkript, and click Edit.
Click the test button, and use label "alertname" with value <value of your alert>:
This will send a test alert to unSkript.
Last updated