Manage Secrets in Grafana Configuration
Keep passwords and tokens out of Git using environment variables, files, and encrypted manifests.
Configuration as code only works if the repository can be shared, and a repository that contains passwords can’t be. Grafana gives you several ways to keep secrets out of the files while still defining everything declaratively. This guide shows where secrets appear, how to inject them, and what to do about the secrets Grafana stores itself.
Where secrets show up
- Server settings: the admin password, the database password, SMTP credentials, OAuth client secrets, and the
secret_key. - Data sources: passwords, tokens, and TLS keys in
secureJsonData. - Contact points: webhook URLs, API keys, and routing keys in
settings. - Provisioning for other tools: Terraform, the Operator, and Git Sync each need an API token or a Git credential.
Method 1: environment variables
The simplest option. Provisioning YAML supports $VAR and ${VAR} in values, and grafana.ini supports ${VAR} and $__env{VAR}:
# provisioning/datasources/postgres.yaml
datasources:
- name: Postgres
uid: postgres-main
type: postgres
url: $POSTGRES_HOST:5432
secureJsonData:
password: $POSTGRES_PASSWORD
# grafana.ini
[security]
admin_password = ${GF_ADMIN_PASSWORD}
Rules to remember:
- Interpolation works on values only, not keys.
- Use
$$when you need a literal dollar sign (for example in a template like{{ $$labels }}). - It does not work inside dashboard JSON, and it is not applied in some alerting fields: rule annotations, query models,
relativeTimeRange, mute timing names and intervals, and notification templates.
In Docker Compose, read variables from an untracked .env file or from your CI secret store. In Kubernetes, map a Secret into the container (see below).
Method 2: secrets as files
Grafana reads a file’s contents into a setting with $__file{}, which trims surrounding whitespace:
[database]
password = $__file{/run/secrets/grafana_db_password}
[security]
secret_key = $__file{/run/secrets/grafana_secret_key}
This suits Docker secrets and Kubernetes Secrets mounted as files, and it avoids leaking values through docker inspect or process environments. The official Docker image also accepts a __FILE suffix on GF_* variable names:
services:
grafana:
image: grafana/grafana:13.1.0
environment:
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
secrets:
- grafana_admin_password
secrets:
grafana_admin_password:
file: ./secrets/grafana_admin_password.txt
Method 3: Kubernetes Secrets
Store credentials in a Secret and expose them as environment variables:
apiVersion: v1
kind: Secret
metadata:
name: grafana-secrets
namespace: monitoring
type: Opaque
stringData:
admin-password: change-me
slack-webhook-url: https://hooks.slack.com/services/XXX/YYY/ZZZ
postgres-password: change-me-too
env:
- name: SLACK_WEBHOOK_URL
valueFrom:
secretKeyRef:
name: grafana-secrets
key: slack-webhook-url
The Grafana Helm chart can take an existing Secret for the admin user and mount others for environment variables, which the Kubernetes guide shows. Don’t commit that Secret as plain YAML. Encrypt it, or generate it from a secret manager:
- SOPS with Flux or Argo CD: commit
grafana-secrets.yamlencrypted, and let the GitOps controller decrypt it in the cluster. The Flux series in this library has a SOPS guide. - External Secrets Operator or a CSI driver: sync secrets from Vault, AWS Secrets Manager, or another store, and commit only the reference.
The secret_key and stored secrets
Grafana encrypts the secrets it stores in its database (such as data source passwords) with the secret_key setting. Two consequences:
- Set it explicitly and keep it the same across restarts, replicas, and rebuilds. The default value is public, so leaving it unset weakens that encryption.
- Don’t change it carelessly. If it changes, existing encrypted values can’t be decrypted. Because data sources are provisioned from files, re-provisioning with the new key restores them, but secrets entered only in the UI are lost.
Admin password caveat
admin_password only takes effect when Grafana first creates its database. To change it later, reset it through the CLI inside the container or through the API, or recreate the database:
grafana cli admin reset-admin-password 'new-password'
Better still, move people to SSO so that the local admin account is only a break-glass login, and store its password in your secret manager.
Tokens for the tools around Grafana
- Terraform and the Operator authenticate with a service account token. Create the service account with a minimal role, store the token in your CI or cluster secret store, and rotate it on a schedule.
- Git Sync needs a Git provider credential: a personal access token, or a GitHub App or OAuth app connection. Pass the token in the
secureblock of theRepositoryresource, so it is stored as a secret rather than in the spec. - The admin provisioning API accepts only basic authentication with an admin user, not API tokens, so restrict who can reach those endpoints.
Checklist
- No plaintext secret in Git history. If one slipped in, rotate it, don’t just delete the commit.
secret_keyset explicitly and stored in your secret manager.- Each environment has its own database password, admin password, and API tokens.
.envand secrets folders are in.gitignore, and CI scans commits for secrets.
Popular Guides
-
Operate Grafana as Code: The Big Picture What can live in files, what needs an API, and how the pieces fit together. -
Configure the Grafana Server with grafana.ini and Environment Variables Know where config files live, how overrides are ordered, and how to keep settings in Git. -
Provision Dashboards from JSON Files Load dashboards from disk, map folders from directories, and author them without losing changes. -
Provision Alert Rules, Contact Points, and Notification Policies Define the whole alerting pipeline in YAML so it is reviewed and versioned like code. -
Deploy Grafana on Kubernetes with Helm and Provisioning Values Run Grafana from a Helm chart where data sources, dashboards, and settings all come from values files. -
Reload, Verify, and Troubleshoot Provisioning from the Command Line Find out what Grafana loaded, reload without a restart, and fix the usual provisioning errors.