OpenTelemetry OpenTelemetry

Send Telemetry to Backends and Secure the Pipeline

Configure OTLP exporters for Tempo, Loki, and Prometheus, plus TLS, authentication, and secrets.

14 min Intermediate

Collecting telemetry is half the job; the other half is delivering it to the right place safely. This guide shows how to configure exporters for the common open-source backends (the Grafana stack is used for the examples), and how to secure traffic into and out of the Collector with TLS, authentication, and secrets that stay out of your config files.

OTLP exporters: gRPC and HTTP

Two exporters speak OTLP:

  • otlp uses gRPC. The endpoint is host:port with no scheme, usually port 4317.
  • otlphttp uses HTTP. The endpoint is a URL, usually port 4318, and the exporter appends the signal path (/v1/traces, /v1/metrics, /v1/logs) to it.

Use a name suffix to define several of the same type:

exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc.cluster.local:4317
    tls:
      insecure: true
  otlphttp/loki:
    endpoint: http://loki.observability.svc.cluster.local:3100/otlp
  otlphttp/prometheus:
    endpoint: http://prometheus.observability.svc.cluster.local:9090/api/v1/otlp

Backend specifics worth knowing:

  • Tempo accepts OTLP directly on its receiver ports (4317 gRPC, 4318 HTTP) when the OTLP receiver is enabled in its config.
  • Loki accepts OTLP logs at /otlp on its HTTP port, so with otlphttp the final URL becomes /otlp/v1/logs.
  • Prometheus accepts OTLP metrics at /api/v1/otlp/v1/metrics, but only if it is started with the OTLP receiver enabled (the --web.enable-otlp-receiver flag).

Confirm these paths in each product’s documentation for the version you run. Then route each signal to its backend:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/loki]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/prometheus]

A pipeline may have several exporters to send the same data to more than one place, for example a primary backend and an archive.

TLS on exporters

For anything crossing an untrusted network, use TLS:

exporters:
  otlp/backend:
    endpoint: otlp.example.com:4317
    tls:
      ca_file: /etc/otel/certs/ca.crt        # to trust a private CA
      cert_file: /etc/otel/certs/client.crt  # for mutual TLS
      key_file: /etc/otel/certs/client.key

With a public CA you can omit the whole tls block, since the system roots are used. insecure: true disables TLS and is for in-cluster development only. Avoid insecure_skip_verify: true in production, because it removes the protection TLS provides.

TLS and authentication on receivers

A Collector that listens on a network port should require TLS and, if it’s reachable by anything you don’t control, authentication.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/otel/certs/server.crt
          key_file: /etc/otel/certs/server.key
          client_ca_file: /etc/otel/certs/ca.crt   # require client certificates (mTLS)

For token-based authentication, use an authentication extension. The bearertokenauth extension can both send and check a token:

extensions:
  bearertokenauth/client:
    scheme: Bearer
    token: ${env:BACKEND_TOKEN}

exporters:
  otlphttp/backend:
    endpoint: https://otlp.example.com
    auth:
      authenticator: bearertokenauth/client

service:
  extensions: [bearertokenauth/client]

To validate incoming requests, other extensions exist, such as basicauth and oidc, referenced from the receiver with auth: { authenticator: <name> }. Available options differ by distribution and version, so check the extension README.

For simple header-based credentials, you can also set headers on an exporter:

exporters:
  otlphttp/saas:
    endpoint: https://otlp.vendor.example.com
    headers:
      api-key: ${env:VENDOR_API_KEY}

Keep secrets out of config

Never write a token or password literally into a config file that lives in Git. Reference an environment variable instead, and supply the variable from a secret store:

exporters:
  otlphttp/backend:
    headers:
      authorization: Bearer ${env:BACKEND_TOKEN}

On Kubernetes, inject the variable from a Secret:

env:
  - name: BACKEND_TOKEN
    valueFrom:
      secretKeyRef:
        name: otel-backend
        key: token

The Secret itself should be encrypted in Git (for example with SOPS, as in the Flux guides) or synced from a secret manager. The Collector also supports a file: provider (${file:/path/to/secret}) that reads a mounted file, which suits secrets mounted as volumes and rotated by the platform. Whichever source you use, confirm that the provider is available in your distribution; the contrib builds include several.

Network exposure

Since the Collector often runs inside the cluster, its default posture matters:

  • Bind deliberately. Recent versions default receivers to localhost, which is safe but unreachable from other containers. Use 0.0.0.0 only for ports that should be reachable, and protect them with a NetworkPolicy, TLS, or authentication.
  • Disable what you don’t use. Remove unused receivers and Service ports so the surface stays small.
  • Debug endpoints are for operators. Keep zpages and pprof on localhost, and never expose them publicly.
  • Redact before export. Use attributes or transform processors to delete credentials, tokens, and personal data before they leave your network (see the production-readiness guide).

A NetworkPolicy that allows only application namespaces to reach the Collector:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-otlp-from-apps
  namespace: observability
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: opentelemetry-collector
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              telemetry: "enabled"
      ports:
        - protocol: TCP
          port: 4317
        - protocol: TCP
          port: 4318

Verify

  1. Run otelcol validate --config=config.yaml to catch YAML and schema mistakes.
  2. Start the Collector and check its logs for connection or TLS errors on the exporters.
  3. Send a test span, then look for it in the backend. If it doesn’t arrive, compare the exporter’s protocol and port with what the backend actually listens on.
  4. Check the Collector’s own metrics for otelcol_exporter_send_failed_* counters, which rise when a backend rejects data.

Popular Guides