Grafana Grafana

Manage Secrets in Grafana Configuration

Keep passwords and tokens out of Git using environment variables, files, and encrypted manifests.

10 min Intermediate

Configuration as code only works if the repository can be shared, and a repository that contains passwords can’t be. Grafana gives you several ways to keep secrets out of the files while still defining everything declaratively. This guide shows where secrets appear, how to inject them, and what to do about the secrets Grafana stores itself.

Where secrets show up

  • Server settings: the admin password, the database password, SMTP credentials, OAuth client secrets, and the secret_key.
  • Data sources: passwords, tokens, and TLS keys in secureJsonData.
  • Contact points: webhook URLs, API keys, and routing keys in settings.
  • Provisioning for other tools: Terraform, the Operator, and Git Sync each need an API token or a Git credential.

Method 1: environment variables

The simplest option. Provisioning YAML supports $VAR and ${VAR} in values, and grafana.ini supports ${VAR} and $__env{VAR}:

# provisioning/datasources/postgres.yaml
datasources:
  - name: Postgres
    uid: postgres-main
    type: postgres
    url: $POSTGRES_HOST:5432
    secureJsonData:
      password: $POSTGRES_PASSWORD
# grafana.ini
[security]
admin_password = ${GF_ADMIN_PASSWORD}

Rules to remember:

  • Interpolation works on values only, not keys.
  • Use $$ when you need a literal dollar sign (for example in a template like {{ $$labels }}).
  • It does not work inside dashboard JSON, and it is not applied in some alerting fields: rule annotations, query models, relativeTimeRange, mute timing names and intervals, and notification templates.

In Docker Compose, read variables from an untracked .env file or from your CI secret store. In Kubernetes, map a Secret into the container (see below).

Method 2: secrets as files

Grafana reads a file’s contents into a setting with $__file{}, which trims surrounding whitespace:

[database]
password = $__file{/run/secrets/grafana_db_password}

[security]
secret_key = $__file{/run/secrets/grafana_secret_key}

This suits Docker secrets and Kubernetes Secrets mounted as files, and it avoids leaking values through docker inspect or process environments. The official Docker image also accepts a __FILE suffix on GF_* variable names:

services:
  grafana:
    image: grafana/grafana:13.1.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
    secrets:
      - grafana_admin_password

secrets:
  grafana_admin_password:
    file: ./secrets/grafana_admin_password.txt

Method 3: Kubernetes Secrets

Store credentials in a Secret and expose them as environment variables:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-secrets
  namespace: monitoring
type: Opaque
stringData:
  admin-password: change-me
  slack-webhook-url: https://hooks.slack.com/services/XXX/YYY/ZZZ
  postgres-password: change-me-too
env:
  - name: SLACK_WEBHOOK_URL
    valueFrom:
      secretKeyRef:
        name: grafana-secrets
        key: slack-webhook-url

The Grafana Helm chart can take an existing Secret for the admin user and mount others for environment variables, which the Kubernetes guide shows. Don’t commit that Secret as plain YAML. Encrypt it, or generate it from a secret manager:

  • SOPS with Flux or Argo CD: commit grafana-secrets.yaml encrypted, and let the GitOps controller decrypt it in the cluster. The Flux series in this library has a SOPS guide.
  • External Secrets Operator or a CSI driver: sync secrets from Vault, AWS Secrets Manager, or another store, and commit only the reference.

The secret_key and stored secrets

Grafana encrypts the secrets it stores in its database (such as data source passwords) with the secret_key setting. Two consequences:

  • Set it explicitly and keep it the same across restarts, replicas, and rebuilds. The default value is public, so leaving it unset weakens that encryption.
  • Don’t change it carelessly. If it changes, existing encrypted values can’t be decrypted. Because data sources are provisioned from files, re-provisioning with the new key restores them, but secrets entered only in the UI are lost.

Admin password caveat

admin_password only takes effect when Grafana first creates its database. To change it later, reset it through the CLI inside the container or through the API, or recreate the database:

grafana cli admin reset-admin-password 'new-password'

Better still, move people to SSO so that the local admin account is only a break-glass login, and store its password in your secret manager.

Tokens for the tools around Grafana

  • Terraform and the Operator authenticate with a service account token. Create the service account with a minimal role, store the token in your CI or cluster secret store, and rotate it on a schedule.
  • Git Sync needs a Git provider credential: a personal access token, or a GitHub App or OAuth app connection. Pass the token in the secure block of the Repository resource, so it is stored as a secret rather than in the spec.
  • The admin provisioning API accepts only basic authentication with an admin user, not API tokens, so restrict who can reach those endpoints.

Checklist

  • No plaintext secret in Git history. If one slipped in, rotate it, don’t just delete the commit.
  • secret_key set explicitly and stored in your secret manager.
  • Each environment has its own database password, admin password, and API tokens.
  • .env and secrets folders are in .gitignore, and CI scans commits for secrets.

Popular Guides