When everything is defined in files, the failures also live in files: a bad indent, a missing UID, a path that doesn’t exist in the container. Grafana tells you what it did, but you have to know where to look. This guide covers the reload endpoints, a few verification commands, and the errors you will meet most often, so you can fix them without opening the UI.

What gets applied when

Resource	When it is read	How to re-apply
grafana.ini and GF_* env vars	Startup only	Restart Grafana
Data sources	Startup	Restart, or the reload endpoint
Dashboards	Startup, then every updateIntervalSeconds	Wait, or the reload endpoint
Alerting resources	Startup	Restart (or re-run provisioning through the Admin API as documented for your version)
Plugin app settings	Startup	Restart, or the reload endpoint

Reload without a restart

These admin endpoints re-read provisioning files. They use POST and basic authentication with a Grafana admin user; API tokens don’t work for the Admin API:

BASE=http://localhost:3000
AUTH="admin:$GRAFANA_ADMIN_PASSWORD"

curl -X POST -u "$AUTH" $BASE/api/admin/provisioning/dashboards/reload
curl -X POST -u "$AUTH" $BASE/api/admin/provisioning/datasources/reload
curl -X POST -u "$AUTH" $BASE/api/admin/provisioning/plugins/reload
curl -X POST -u "$AUTH" $BASE/api/admin/provisioning/notifications/reload

A successful call returns a JSON message saying the configuration was reloaded. Don’t expose these endpoints publicly, because they need admin credentials. For anything else, restarting is always safe.

Check what Grafana actually loaded

Compare the files with the API’s view:

# health
curl -s $BASE/api/health

# effective server settings (sensitive values masked)
curl -s -u "$AUTH" $BASE/api/admin/settings | jq '.paths, .server'

# data sources, with read-only flag for provisioned ones
curl -s -u "$AUTH" $BASE/api/datasources | jq '.[] | {name, uid, type, readOnly}'

# data source connectivity
curl -s -u "$AUTH" $BASE/api/datasources/uid/prometheus/health

# dashboards and folders
curl -s -u "$AUTH" "$BASE/api/search?type=dash-db" | jq '.[] | {title, uid, folderTitle}'
curl -s -u "$AUTH" $BASE/api/folders | jq '.[] | {title, uid}'

# alerting
curl -s -u "$AUTH" $BASE/api/v1/provisioning/alert-rules | jq '.[] | {uid, title}'
curl -s -u "$AUTH" $BASE/api/v1/provisioning/contact-points | jq '.[].name'
curl -s -u "$AUTH" $BASE/api/v1/provisioning/policies | jq .

If something you provisioned is missing from these lists, the logs will say why.

Read the logs

Provisioning messages come from loggers whose names start with provisioning. Raise verbosity while debugging:

# docker compose
GF_LOG_LEVEL=debug docker compose up -d grafana
docker compose logs grafana | grep -i provision

# kubernetes
kubectl logs deploy/grafana -n monitoring -c grafana | grep -i provision

Look for the file name in the message. Grafana names the file it failed to parse or the resource it rejected.

Check files before they reach Grafana

Most errors can be caught in CI:

# YAML syntax
yamllint grafana/provisioning

# JSON syntax and required fields
jq empty dashboards/**/*.json
jq -e '.uid and .title' dashboards/*/*.json >/dev/null

# duplicate dashboard UIDs
jq -r '.uid' dashboards/*/*.json | sort | uniq -d

# every provisioned file starts with apiVersion: 1
grep -L '^apiVersion: 1' grafana/provisioning/*/*.yaml

A cheap end-to-end test is to start a throwaway Grafana from the same files in CI, wait for /api/health, and assert on the API output above.

Common problems and fixes

A data source or dashboard doesn’t appear. The provisioning folder isn’t mounted where Grafana looks. Confirm that [paths] provisioning (default /etc/grafana/provisioning) matches your mount, and that the subfolder names are exactly datasources, dashboards, alerting, and plugins.

“Permission denied” reading files. The official image runs as a non-root user, so mounted files must be readable by it. Make them world-readable (chmod -R a+rX), or set matching ownership on the volume.

Dashboard fails with “Dashboard title cannot be empty”. The JSON is wrapped in the API envelope ({"dashboard": {...}}). Provisioned files must contain the dashboard model itself.

Dashboard loads but panels say data source not found. The panel references a UID that isn’t provisioned, or the file still has an ${DS_...} placeholder from an export. Set uid in the data source file, and use the same UID in the dashboard JSON.

Two dashboards fight, or one keeps reverting. They share a uid, or the same title in the same folder. Make every UID unique across the installation. If a UI edit keeps being overwritten, that is the file winning, which is by design unless the provider has allowUiUpdates: true.

Removed a file but the dashboard is still there. The provider has disableDeletion: true, or the dashboard was moved into a folder by hand. Delete it through the API, or turn off disableDeletion.

Data source change ignored on a multi-instance setup. With several Grafana replicas, an instance only updates a data source if the file’s version is the same or higher than the stored one. Increase version when you change the file.

Environment variable not substituted. Interpolation applies to values, not keys. It doesn’t work in dashboard JSON, alert rule annotations, query models, mute timings, or templates. Check that the variable is actually set in the container (docker compose exec grafana env | grep NAME).

Alerting import conflicts. A resource with the same UID or name exists from a manual edit. Delete the hand-made one, or add it to a delete... block in the file, then restart.

Secrets stopped working after a rebuild. The secret_key changed, so stored secrets can’t be decrypted. Restore the original key, or re-provision the data sources that need credentials.

Admin password didn’t change. admin_password applies only on first database creation. Use grafana cli admin reset-admin-password or recreate the volume.

A Helm or Operator resource isn’t syncing. For the chart, check the sidecar’s container logs for label or namespace mismatches. For the Operator, run kubectl describe on the resource and read its status conditions.

A reliable change routine

1. Change the file in a branch.
2. CI runs the linters above and starts a disposable Grafana from the files.
3. Merge. The delivery mechanism restarts or reconciles Grafana.
4. Check /api/health and the verification commands.
5. If it misbehaves, revert the commit. Because the files are the truth, reverting restores the previous state.