unSkript Documentation
Free SandboxBlogGitHubSlack Community
  • What is unSkript?
  • Guides
    • Contribute to Open Source
    • Getting started
      • Sign Up/Install
      • Authentication
        • Okta configuration
          • Okta Groups Sync
      • Create a Proxy
      • Add Credentials to Connect your Resources
      • Key Terms
  • RunBooks
    • Prebuilt xRunBooks
      • xRunBooks for CloudOps
      • xRunBooks for Cost Optimization
      • xRunBooks for DevOps
      • xRunBooks for IAM
      • xRunBooks for SecOps
      • xRunBooks for SRE
      • xRunBooks for Troubleshooting
    • Creating RunBooks
      • Importing unSkript xRunBooks
      • Create a xRunBook
      • Add an Action
      • Connecting Actions
      • Create a RunBook Input Parameter
      • Create a Credential
      • Writing Notes
      • Running XRunBooks
        • Running RunBooks with Approvals
      • Schedules
      • xRunBook Executions
      • Alarms
        • Create an Alarm Webhook
          • Webhooks for onprem installations
        • Create a Grafana Alert
        • Attaching runbooks to alarms
      • Folders
      • RunBook Outputs
  • Actions
    • What is an Action?
    • Create Custom Actions
    • Create Actions with AI
    • Actions
      • Action Configuration
        • Configure Action Inputs
        • Add Action Inputs
        • Action Output
        • Action Iterator
          • Using Iterators
        • Action Poll
        • Action Start Condition
      • Anatomy of an Action
    • Prebuilt Actions
  • Healthchecks
    • What is a HealthCheck?
    • Create a HealthCheck
      • Create a HealthCheck Folder
      • Adding Checks
    • HealthCheck Parameters
    • Running HealthChecks
    • HealthCheck Remediation
  • Workflows
    • What is a Workflow?
  • Connnecting
    • Connectors
      • Airflow
        • Airflow Actions
      • AWS
        • AWS Actions
          • AWS Cloudwatch Actions
          • AWS EC2 Actions
          • AWS ECS Actions
          • AWS EKS Actions
          • AWS ELB Actions
          • AWS IAM Actions
          • AWS Postgres Actions
          • AWS RDS Actions
          • AWS RedShift Actions
          • AWS S3 Actions
          • AWS VPC Actions
      • Azure
        • Azure Actions
      • ChatGPT
        • ChatGPT Actions
      • Datadog
        • Configuring webhook in Datadog
        • Datadog Actions
          • Datadog Alert Actions
          • Datadog Metrics Actions
          • Datadog Monitor Actions
      • Elasticsearch
        • Elasticsearch Actions
      • GCP
        • GCP Actions
          • GCP Bucket Actions
          • GCP GKE Actions
          • GCP IAM Actions
          • GCP VM Actions
      • Github
        • Github Actions
      • Grafana
        • Grafana Actions
      • Hadoop
        • Hadoop Actions
      • Jenkins
        • Jenkins Actions
      • Jira
        • Jira Actions
      • Kafka
        • Kafka Actions
      • Kubernetes
        • K8s Actions
      • MongoDB
        • MongoDB Actions
      • MS SQL
        • MSSQL Actions
      • MySQL
        • MySQL Actions
      • Netbox
        • Netbox Actions
      • Nomad
        • Nomad Actions
      • OpenSearch
        • opensearch Actions
      • Pingdom
        • Pingdom Actions
      • Postgres
        • Postgres Actions
      • Prometheus
        • Prometheus Actions
      • Redis
        • Redis Actions
      • REST
        • REST Actions
      • SalesForce
        • SalesForce Actions
      • Slack
        • Slack Actions
      • Snowflake
        • Snowflake Actions
      • Splunk
        • Splunk Actions
      • SSH
        • SSH Actions
      • Stripe
        • Stripe Actions
      • Terraform
        • Terraform Actions
      • Zabbix
    • Proxies
      • Environment
        • RunBooks Across Environments: ServiceIDs
      • unSkript Proxy
      • AWS Proxy
      • GCP Proxy
    • Secret store
      • Vault
  • Tooling
    • Role Based Access Control
      • RBAC Roles
    • API reference
      • Authentication
      • Endpoints
        • Executions
        • RunBooks
        • Schedules
    • Command Line Tool
    • Notifications
  • Fundamentals
    • Jupyter Notebook 101
  • Use Cases
    • DevOps
    • SRE
  • Open source
    • Runbooks.sh
    • Contribute to Open Source
  • Lists
    • AWS Service Quota list
Powered by GitBook
On this page
  • Creating Grafana Rule
  • Create the Contact Point
  • Notification Policy
  • Test the alarm

Was this helpful?

  1. RunBooks
  2. Creating RunBooks
  3. Alarms

Create a Grafana Alert

In this document, we'll walk through the steps to create an alert with Grafana.

PreviousWebhooks for onprem installationsNextAttaching runbooks to alarms

Last updated 1 year ago

Was this helpful?

Use case: When the Grafana alert fires, we would like to run an unSkript RunBook to remediate the situation.

The high level steps for creating a Grafana alert:

    1. These are the parameters that will cause the alert to fire.

  1. into unSkript

    1. The contact information - where should the alert fire?

    1. How often will the alert notify unSkript?

    1. Sent test data to unSkript.

Each of these high level steps are described in greater detail in this page:

Creating Grafana Rule

First we must create the alert at Grafana. If you already have the alert you wish to use, you can skip this section.

On this page, we will create a new rule.

  1. Give your Rule a name, and rule type ('Grafana managed alert')

  1. Create alert queries.

  2. The second query is going to take the results from the first query, and if the last value is over 200, the alert will fire. (There are more than 200 stars, so this alert will begin firing immediately.)

  3. Define the alert conditions. In the screenshot below, this will check every 1 minute, and begin firing the error after 5 minutes.

Note: This frequency is much too fast this alert. This frequency will be useful while provisioning the alert. We will update the conditions once the alert is set.

Step 4 of the process is to add details. Add any details that you wish, and the save the alert.

Create the Contact Point

Next, we need to connect the alert to unSkript.

  1. In the Grafana dashboard, we need to create a contact point. This can be found in the alarms menu in the left navigation:

  2. Create a new Contact point.

    1. Type: Webhook

    2. url: The webhook url provided by unSkript

    3. HTTP method: POST

    4. Username (from unskript webhook details)

    5. Password (from unSkriupt webhook details)

  3. Save your contact point.

Notification Policy

Ok, so we've created an alert, and a connection to unSkript. Now let's build a notification policy.

Grafana is set with a default notification policy, or you can set an individual policy for specific alerts to be sent to unSkript.

Let's create a specific routing for this alert:

  1. Matching rules: Select the alert. Set label to "alertname" and the Value to the name of your alert (in the screenshot below, the alert name is "Docs_rule.")

  2. Contact Point: Select the Contact point you created with unSkript webhook credentials.

  3. Override general settings. These settings will force the alarm to fire every 2 minutes. That is very frequent, but useful when setting up your alerts. Choose settings that make sense for your alert.

Test the alarm

Send a test alarm message. In the Connection Points menu, select the connection pointing to unSkript, and click Edit.

Click the test button, and use label "alertname" with value <value of your alert>:

This will send a test alert to unSkript.

To create a Grafana alert, log into your Grafana dashboard, and in the left navigation, click the alarm bell, and choose alert rules:

This query is reading a Postgres database for the count of Github stars for (click the link, and give us a star!)

If you have not yet created a Grafana webhook, with one of your Grafana Credentials.

Awesome-Cloudops-Automation
Create a Webhook
Creating a Grafana rule
Creating a Grafana Contact point
Notification Policy
Test the Alarm
Defining the rule type
Creating the query
refining the query for alert
Defining alert conditions
Adding a contact point
Inserting the webhook details
Adding a Notification Policy
An example policy
Testing the alarm