Create a Grafana Alert

In this document, we'll walk through the steps to create an alert with Grafana.

Use case: When the Grafana alert fires, we would like to run an unSkript RunBook to remediate the situation.

The high level steps for creating a Grafana alert:

1. Creating a Grafana rule

   1. These are the parameters that will cause the alert to fire.

2. Creating a Grafana Contact point into unSkript

   1. The contact information - where should the alert fire?

3. Notification Policy

   1. How often will the alert notify unSkript?

4. Test the Alarm

   1. Sent test data to unSkript.

Each of these high level steps are described in greater detail in this page:

Creating Grafana Rule

First we must create the alert at Grafana. If you already have the alert you wish to use, you can skip this section.

To create a Grafana alert, log into your Grafana dashboard, and in the left navigation, click the alarm bell, and choose alert rules:

On this page, we will create a new rule.

1. Give your Rule a name, and rule type ('Grafana managed alert')

Defining the rule type

1. Create alert queries.

2. This query is reading a Postgres database for the count of Github stars for Awesome-Cloudops-Automation (click the link, and give us a star!)

   

   Creating the query
3. The second query is going to take the results from the first query, and if the last value is over 200, the alert will fire. (There are more than 200 stars, so this alert will begin firing immediately.)

   

   refining the query for alert
4. Define the alert conditions. In the screenshot below, this will check every 1 minute, and begin firing the error after 5 minutes.

Note: This frequency is much too fast this alert. This frequency will be useful while provisioning the alert. We will update the conditions once the alert is set.

Defining alert conditions

Step 4 of the process is to add details. Add any details that you wish, and the save the alert.

Create the Contact Point

Next, we need to connect the alert to unSkript.

1. If you have not yet created a Grafana webhook, Create a Webhook with one of your Grafana Credentials.

2. In the Grafana dashboard, we need to create a contact point. This can be found in the alarms menu in the left navigation:

3. 

   Adding a contact point

4. Create a new Contact point.

   1. Type: Webhook

   2. url: The webhook url provided by unSkript

   3. HTTP method: POST

   4. Username (from unskript webhook details)

   5. Password (from unSkriupt webhook details)

5. 

   Inserting the webhook details
6. Save your contact point.

Notification Policy

Ok, so we've created an alert, and a connection to unSkript. Now let's build a notification policy.

Adding a Notification Policy

Grafana is set with a default notification policy, or you can set an individual policy for specific alerts to be sent to unSkript.

Let's create a specific routing for this alert:

1. Matching rules: Select the alert. Set label to "alertname" and the Value to the name of your alert (in the screenshot below, the alert name is "Docs_rule.")

2. Contact Point: Select the Contact point you created with unSkript webhook credentials.

3. Override general settings. These settings will force the alarm to fire every 2 minutes. That is very frequent, but useful when setting up your alerts. Choose settings that make sense for your alert.

An example policy

Test the alarm

Send a test alarm message. In the Connection Points menu, select the connection pointing to unSkript, and click Edit.

Click the test button, and use label "alertname" with value <value of your alert>:

Testing the alarm

This will send a test alert to unSkript.