Istio Istio

Observe Your Mesh: Prometheus, Grafana, Tracing, and Kiali

Every sidecar is already emitting metrics and traces — install the add-ons that turn that into something you can actually look at.

12 min Intermediate

Every Envoy sidecar in a mesh is already generating detailed metrics and trace data as a side effect of handling traffic; nothing extra needs to be added to your applications for that data to exist. What’s missing without additional components is somewhere to send it, store it, and look at it. This guide installs the pieces that do that and get a mesh’s traffic actually visible.

What each add-on is for

  • Prometheus scrapes and stores the metrics every sidecar exposes: request rates, error rates, latencies, per service and per route. This library’s dedicated Prometheus guides cover the tool itself in depth; here it’s specifically the backend Istio’s own dashboards query.
  • Grafana visualizes that Prometheus data through pre-built Istio dashboards: mesh-wide traffic overview, per-service detail, control plane health.
  • A distributed tracing backend (Jaeger has been the longest-standing choice in Istio’s sample add-ons; some environments use Zipkin or an OpenTelemetry-based backend instead) collects traces showing a single request’s full path across every service it touched, which is what makes it possible to see where in a multi-hop request a slowdown actually occurred, rather than only knowing it occurred somewhere.
  • Kiali is a service mesh console built specifically for Istio: a live topology graph of how services actually talk to each other, backed by the same metrics and trace data, plus the ability to inspect and in some cases edit mesh configuration directly from its UI.

Install the sample add-ons

Istio’s release bundle includes ready-to-apply manifests for all of these under samples/addons:

cd istio-*  # the directory from installing istioctl
kubectl apply -f samples/addons
kubectl get pods -n istio-system

You should see Pods for prometheus, grafana, kiali, and a tracing backend start up alongside istiod and any gateways already running in istio-system. These sample manifests are meant for evaluation and learning, not as a production-ready deployment of each tool: default resource limits, storage, and retention are all modest, and a real deployment would typically run Prometheus and Grafana with their own dedicated, properly sized configuration rather than the bundled sample.

Reach Kiali

None of these add-ons are exposed outside the cluster by default. Port-forward to reach Kiali’s UI:

kubectl port-forward svc/kiali -n istio-system 20001:20001

Open http://localhost:20001. With traffic flowing through meshed services, Kiali’s graph view renders live: which services are talking to which, color-coded by health, with request volume and error rate visible per connection.

Why the graph might look wrong, or empty

If Kiali’s topology view is empty, sparse, or shows disconnected nodes that should clearly be talking to each other, the most common cause is the same one covered in the previous guide: a missing app label on the underlying Deployments and Services. Kiali builds its graph from that label; workloads without it exist in the mesh and carry traffic just fine, but Kiali has no reliable way to identify and connect them in its visualization. Check for the label before assuming the mesh itself is misconfigured.

The second most common cause is simply no traffic yet: Kiali’s graph reflects observed traffic over a selected time window, so a freshly meshed service with no requests sent to it yet has nothing to display. Send a few requests through and refresh.

Grafana’s pre-built dashboards

Port-forward to Grafana the same way:

kubectl port-forward svc/grafana -n istio-system 3000:3000

The sample installation ships with Istio-specific dashboards already provisioned, mesh-wide traffic, per-service detail, control plane performance, rather than requiring you to build them from scratch. These are a genuinely good starting point for understanding what metrics Istio exposes and how they’re structured; the Grafana guide elsewhere in this library covers customizing and extending dashboards like these once you’re past exploring the defaults.

Treat the sample install as a starting point, not a destination

Once you’ve confirmed the mesh is observable and you understand what each tool shows, the natural next step for anything beyond local evaluation is replacing the bundled samples with properly configured, independently managed installations, likely using the dedicated guides for Prometheus and Grafana elsewhere in this library, rather than relying on samples/addons indefinitely. The sample manifests exist to get you to “I can see my mesh” quickly, not to be the long-term observability stack for a production cluster.

Popular Guides