Grafana alerting has four moving parts: alert rules decide when something is wrong, contact points say where to send a notification, notification policies route alerts to contact points, and mute timings and templates shape when and how messages go out. All four can be provisioned from YAML in the provisioning/alerting folder, which makes alerting reviewable instead of a pile of UI settings. This guide builds a small working pipeline.

Provisioned alerting resources can’t be edited in the UI, and importing a resource that already exists causes a conflict, so remove hand-made duplicates first. Every file starts with apiVersion: 1.

Contact points

provisioning/alerting/contact-points.yaml:

apiVersion: 1

contactPoints:
  - orgId: 1
    name: team-slack
    receivers:
      - uid: team-slack-1
        type: slack
        disableResolveMessage: false
        settings:
          url: ${SLACK_WEBHOOK_URL}
          recipient: '#alerts'

  - orgId: 1
    name: oncall-email
    receivers:
      - uid: oncall-email-1
        type: email
        settings:
          addresses: oncall@example.com

Notes:

• uid on each receiver must be unique and stable. Grafana uses it to update the receiver in place.
• type is the integration, such as slack, email, webhook, pagerduty, opsgenie, or prometheus-alertmanager. The keys under settings depend on the type, so check the documentation for yours.
• Secrets come from environment variables with $VAR or ${VAR}, so the webhook URL never lives in Git. Use $$ to keep a literal dollar sign.
• To remove one, add deleteContactPoints with its orgId and uid.

Notification policies

The policy tree decides which contact point handles which alert. The whole tree is one resource, and provisioning it overwrites all existing policies.

provisioning/alerting/policies.yaml:

apiVersion: 1

policies:
  - orgId: 1
    receiver: team-slack            # default for anything not matched below
    group_by: ['alertname', 'namespace']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: oncall-email
        object_matchers:
          - ['severity', '=', 'critical']
        repeat_interval: 1h
      - receiver: team-slack
        object_matchers:
          - ['team', '=', 'platform']
        mute_time_intervals:
          - weekends

• receiver at the top is the fallback contact point.
• group_by batches related alerts into one notification.
• group_wait, group_interval, repeat_interval control how quickly the first message goes out, how often updates are sent for a group, and how often an unresolved alert is repeated.
• routes are child policies using the same fields. The first matching route wins unless you set continue: true.
• object_matchers are [label, operator, value] triples. The operators are =, !=, =~, and !~.

To reset the tree to Grafana’s default, use resetPolicies: [1] with the organization ID.

Mute timings

apiVersion: 1

muteTimes:
  - orgId: 1
    name: weekends
    time_intervals:
      - weekdays: ['saturday', 'sunday']
        location: 'UTC'

times (with start_time and end_time), weekdays, months, years, and days_of_month can be combined, and ranges use colons, such as 'monday:friday'. Delete with deleteMuteTimes. Mute timings can’t use variable interpolation in their names or intervals.

Notification templates

apiVersion: 1

templates:
  - orgId: 1
    name: slack_title
    template: |
      {{ define "slack_title" }}
        [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}
      {{ end }}

Reference a template from a contact point’s settings (for example the Slack title field) as {{ template "slack_title" . }}. Template names and content don’t support variable interpolation, so write them literally.

Alert rules

Rules are organized in groups that share an evaluation interval and live in a folder. Each rule has a list of query and expression steps, and a condition naming the step that decides whether it fires.

provisioning/alerting/rules.yaml:

apiVersion: 1

groups:
  - orgId: 1
    name: webapp
    folder: Applications
    interval: 1m
    rules:
      - uid: webapp-down
        title: Webapp targets down
        condition: C
        for: 5m
        noDataState: NoData
        execErrState: Error
        labels:
          severity: critical
          team: platform
        annotations:
          summary: 'No healthy webapp targets for 5 minutes'
          runbook_url: https://runbooks.example.com/webapp-down
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              refId: A
              expr: sum(up{job="webapp"})
              instant: true
          - refId: B
            datasourceUid: __expr__
            model:
              refId: B
              type: reduce
              expression: A
              reducer: last
              datasource:
                type: __expr__
                uid: __expr__
          - refId: C
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: lt
                    params: [1]
              datasource:
                type: __expr__
                uid: __expr__

How to read it:

• Step A queries Prometheus (by the data source uid you provisioned) for the number of healthy targets over the last ten minutes. relativeTimeRange is in seconds.
• Step B reduces the result to a single value. Expressions use the special data source UID __expr__.
• Step C is a threshold: it fires when the value is less than 1. condition: C marks it as the deciding step.
• for: 5m requires the condition to hold for five minutes before the alert fires, which avoids flapping.
• noDataState and execErrState say what to do when the query returns nothing or fails: NoData, Alerting, or OK for the first, and Error, Alerting, or OK for the second.
• labels feed the notification policy tree, so severity: critical is what sent this alert down the oncall-email route above.
• uid (up to 40 characters of letters, digits, -, and _) must be unique and stable. Optionally add dashboardUid and panelId to link a rule to a panel.
• Folder: use a folder that exists. The simplest way is to reuse the title of a folder that your dashboard provider creates.

Variable interpolation does not work in rule annotations, relativeTimeRange, the query model, or template content, so write those values out.

To delete a rule, add deleteRules with its orgId and uid:

apiVersion: 1

deleteRules:
  - orgId: 1
    uid: webapp-down

Apply and verify

Alerting resources are read at startup, so restart Grafana after changing the files:

docker compose restart grafana

curl -s -u admin:$GRAFANA_ADMIN_PASSWORD \
  http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[] | {uid, title, folderUID}'
curl -s -u admin:$GRAFANA_ADMIN_PASSWORD \
  http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
curl -s -u admin:$GRAFANA_ADMIN_PASSWORD \
  http://localhost:3000/api/v1/provisioning/policies | jq .receiver

Then test delivery by temporarily lowering a threshold, or by sending a test notification from the contact point’s API.

Writing rules faster

Rule YAML is verbose. Build a rule once in a scratch instance, then export it in provisioning format through the alerting provisioning API (the export routes under /api/v1/provisioning/, with ?format=yaml), and commit the result with a stable uid. For many similar rules, generate the YAML from a template in CI, or manage them with Terraform or the Grafana Operator, which both have alert rule resources.