Grafana Grafana

Manage Grafana with the Grafana Operator

Declare instances, folders, dashboards, data sources, and alert rules as Kubernetes custom resources.

15 min Advanced

The Grafana Operator turns each Grafana resource into a Kubernetes object. A GrafanaDashboard is a manifest, a GrafanaDatasource is a manifest, and the operator reconciles them against a Grafana instance through its API. That fits GitOps tools like Flux and Argo CD naturally, and it works for Grafana instances inside or outside the cluster, including Grafana Cloud. All resources use apiVersion: grafana.integreatly.org/v1beta1.

Install the operator

Install it with Helm from its OCI chart (check the operator’s installation docs for the current chart location and version, and pin it):

helm upgrade -i grafana-operator \
  oci://ghcr.io/grafana/helm-charts/grafana-operator \
  --namespace grafana --create-namespace \
  --version <operator-version>

The operator’s docs also describe a Kustomize installation. With Flux, put the same chart in a HelmRelease.

Describe the instance

An instance can be managed (the operator deploys Grafana) or external (it connects to one that already exists). A managed instance:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  namespace: grafana
  labels:
    dashboards: "grafana"
spec:
  config:
    log:
      mode: console
    security:
      admin_user: root
      admin_password: secret     # example only; see below

An external instance, such as a Grafana Cloud stack, points at a URL and takes a token from a Secret:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: my-stack
  namespace: grafana
  labels:
    dashboards: my-stack
spec:
  external:
    url: https://my-stack.grafana.net/
    apiKey:
      name: grafana-cloud-credentials
      key: GRAFANA_CLOUD_INSTANCE_TOKEN

The spec.config block takes the same sections and keys as grafana.ini. Don’t put a real password in a manifest. The operator documentation shows how to read admin credentials from a Secret.

The labels on the instance matter: every other resource selects its target instance with instanceSelector.matchLabels, using those labels.

Folders

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: applications
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  title: Applications
  uid: applications     # set a stable UID so other resources can reference it

If title is omitted, it defaults to metadata.name. Set spec.uid whenever other resources will point at the folder with folderRef, because without it the reconciler may adopt an existing folder with the same name.

Data sources

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: prometheus
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  datasource:
    name: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus-service:9090
    isDefault: true
    jsonData:
      timeInterval: 5s

Credentials come from a Secret in the same namespace through valuesFrom, so they never appear in the manifest:

apiVersion: v1
kind: Secret
metadata:
  name: credentials
type: Opaque
stringData:
  PROMETHEUS_USERNAME: grafana
  PROMETHEUS_PASSWORD: change-me
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: prometheus-secured
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  valuesFrom:
    - targetPath: basicAuthUser
      valueFrom:
        secretKeyRef:
          name: credentials
          key: PROMETHEUS_USERNAME
    - targetPath: secureJsonData.basicAuthPassword
      valueFrom:
        secretKeyRef:
          name: credentials
          key: PROMETHEUS_PASSWORD
  datasource:
    name: prometheus-secured
    type: prometheus
    access: proxy
    basicAuth: true
    url: http://prometheus-service:9090
    secureJsonData:
      basicAuthPassword: ${PROMETHEUS_PASSWORD}

Don’t put credentials in the url. Encrypt or sync the Secret with SOPS or External Secrets, as with any Kubernetes Secret.

Dashboards

A GrafanaDashboard can take its JSON from several sources, which is the operator’s biggest strength:

# inline JSON
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: simple
spec:
  resyncPeriod: 30s
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  folderRef: applications
  json: >
    { "uid": "simple", "title": "Simple Dashboard", "panels": [] }
# from a ConfigMap
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  configMapRef:
    name: dashboard-definition
    key: json
# from a URL, such as grafana.com or a raw Git file
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://grafana.com/api/dashboards/1860/revisions/30/download"
# from an OCI artifact
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  oci:
    reference: ghcr.io/team-a/dashboards:v1.4.7
    path: board.json
    pullSecretRef:
      name: ghcr-pull

Other fields you will use:

  • folderRef points at a GrafanaFolder by name, and folderUID at an existing folder’s UID.
  • gzipJson holds the dashboard as base64-encoded gzip (cat dashboard.json | gzip | base64 -w0), for dashboards too large for a manifest.
  • variables overrides template variables, for example to set a namespace or data source per environment.
  • resyncPeriod and contentCacheDuration control how often content is re-applied and how long fetched URLs are cached.

Alert rules

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
  name: webapp-alerts
spec:
  folderRef: applications
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  interval: 5m
  rules:
    - uid: webapp-down
      title: Webapp targets down
      condition: C
      for: 5m
      noDataState: NoData
      execErrState: Error
      data:
        - refId: A
          # the Prometheus query step, as in the alerting guide
        - refId: C
          # the threshold expression step

Every rule group needs a folder and at least one rule. The data steps use the same structure as provisioning files, so you can reuse the YAML from the alerting guide. The operator also has resources for contact points; see its examples for the current list.

Operate it

kubectl get grafanas,grafanafolders,grafanadatasources,grafanadashboards -A
kubectl describe grafanadashboard simple -n grafana

describe shows the resource’s status conditions, which say whether the last sync to Grafana succeeded and why not if it failed. Common causes: an instanceSelector that matches no instance, a wrong token, or a Secret in the wrong namespace.

Operator or file provisioning?

Use the Operator when you already run GitOps on Kubernetes, want each team to own its dashboards and alerts as separate objects, or manage Grafana Cloud or several instances. Use the Helm chart’s provisioning values when you have one Grafana and a handful of resources, since that adds no extra controller. Both can coexist, but let one tool own each resource.

Popular Guides