OpenTelemetry OpenTelemetry

Troubleshoot the Collector and SDKs

A systematic checklist for when telemetry doesn't arrive: debug exporter, health checks, internal metrics, and common mistakes.

12 min Intermediate

“I don’t see my data” is the most common OpenTelemetry problem, and the cause is almost always one of a short list of configuration mistakes. Work from the application outward, one hop at a time, and prove each hop before moving to the next.

App SDK ──▶ Collector receiver ──▶ processors ──▶ exporter ──▶ Backend
   1              2                    3              4            5

Step 0: validate the config

otelcol-contrib validate --config=config.yaml

This catches YAML errors, unknown component names, bad option names, and pipelines that reference undefined components. Use the binary name that matches your distribution (otelcol, otelcol-contrib, otelcol-k8s). In a container, run it with the same image and mounted config as your deployment.

Step 1: is the SDK sending?

Enable SDK diagnostics and a console exporter to separate “the app produced nothing” from “the app produced it but it didn’t arrive”:

export OTEL_LOG_LEVEL=debug
export OTEL_TRACES_EXPORTER=console   # or "logging", depending on the language

Check the usual causes:

  • Endpoint and protocol don’t match. OTEL_EXPORTER_OTLP_PROTOCOL must match the port: grpc goes to 4317, http/protobuf goes to 4318. Sending HTTP to the gRPC port, or the reverse, fails with errors that look like connection or parsing problems.
  • Wrong path with HTTP. OTEL_EXPORTER_OTLP_ENDPOINT is a base URL; the SDK adds /v1/traces. Signal-specific variables such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT are used as given, so they must include the full path.
  • localhost inside a container. The app’s localhost is the app itself. Use the Collector’s Service or container name.
  • The SDK is disabled or the exporter is none. Look for OTEL_SDK_DISABLED=true or OTEL_TRACES_EXPORTER=none.
  • No service.name. Data is accepted but appears as unknown_service, which looks like missing data. Set OTEL_SERVICE_NAME.
  • Short-lived processes exit before the export. Batch exporters send periodically, so call the SDK’s shutdown or flush before exit.

Step 2: is the receiver reachable?

curl -i http://collector:4318/v1/traces \
  -H 'Content-Type: application/json' -d '{}'

An HTTP 200 response (an empty JSON body is fine) shows the HTTP receiver is reachable. For gRPC, a tool such as grpcurl or a TCP check (nc -zv collector 4317) proves the port is open. If it isn’t:

  • The receiver may be bound to localhost. Set endpoint: 0.0.0.0:4317 explicitly.
  • The Kubernetes Service may not expose the port, or the pod’s containerPort is missing.
  • A NetworkPolicy, firewall, or service mesh may block the traffic.

Step 3: is the pipeline wired up?

This is the most common Collector mistake: a component defined but not used. Defining exporters.otlp/tempo does nothing until it appears in service.pipelines.<signal>.exporters. The same goes for receivers, processors, extensions, and connectors.

Check:

  • The component’s name in the pipeline matches its definition exactly, including any suffix (otlp/tempo, not otlp).
  • The signal type matches: a traces pipeline can’t use a receiver or exporter that doesn’t support traces.
  • The extension is listed under service.extensions.
  • In Helm values, remember that lists replace, so a pipeline you partially redefined may have lost the default components.

Step 4: see the data inside the Collector

Add the debug exporter to the pipeline:

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug, otlp/tempo]

Read the Collector’s logs. Seeing spans there proves receivers and processors work, so the problem is downstream. Seeing nothing means data never reached the pipeline or a processor dropped it (a filter processor or the tail sampler is the usual suspect). Remove debug or lower its verbosity when finished, because it logs every item.

Step 5: use the Collector’s own signals

Health check:

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
service:
  extensions: [health_check]

curl http://collector:13133/ returns a status. Use it for Kubernetes liveness and readiness probes.

Internal metrics are served on port 8888 by default (check service.telemetry.metrics for your version):

curl -s http://collector:8888/metrics | grep -E 'otelcol_(receiver|processor|exporter)_'

Read them as a funnel:

Metric familyMeaning
otelcol_receiver_accepted_*Items received successfully
otelcol_receiver_refused_*Rejected, often by memory_limiter
otelcol_processor_dropped_*Dropped by a processor
otelcol_exporter_sent_*Delivered to the backend
otelcol_exporter_send_failed_*Backend rejected or unreachable
otelcol_exporter_queue_size and _capacityQueue depth against its limit

Accepted counts rising with sent counts flat tell you the problem is at the exporter. Refused counts point at memory limits. Metric names have been adjusted between releases, so check the Collector’s internal telemetry documentation for yours.

zPages give a live view of pipelines and recent traces:

extensions:
  zpages:
    endpoint: localhost:55679

Browse to /debug/tracez and /debug/pipelinez (through kubectl port-forward on a cluster). Keep it bound to localhost.

Step 6: is the backend accepting it?

Look at exporter errors in the Collector logs:

  • connection refused or deadline exceeded: wrong endpoint, port, or a network block.
  • certificate signed by unknown authority: add tls.ca_file, or fix the endpoint to use the right host name.
  • 401 or 403: the token or header is missing or wrong. Confirm the environment variable is set in the pod (kubectl exec ... -- env), because an unset ${env:VAR} becomes an empty value.
  • 415 or parsing errors: protocol mismatch, such as otlp (gRPC) pointed at an HTTP port.
  • 404 on HTTP: check the base path the backend expects (Loki uses /otlp, Prometheus /api/v1/otlp).
  • Prometheus rejecting OTLP: its OTLP receiver feature flag isn’t enabled.

Auto-instrumentation checks

For Operator-based injection, confirm that the Instrumentation existed before the pod started, that the annotation is on the pod template, and that the application pod has the init container opentelemetry-auto-instrumentation. Restart the workload after creating or changing the resource. For agent-based setups, confirm the agent flag or environment variable is actually applied (-javaagent:, opentelemetry-instrument, --require).

Quick reference

SymptomFirst thing to check
Nothing in the Collector logsSDK endpoint, protocol, port; localhost in containers
Collector receives data, backend doesn’tExporter in the pipeline; endpoint; auth; protocol
Collector fails to startvalidate output; component missing from the image
Data appears as unknown_serviceSet OTEL_SERVICE_NAME
Memory growth or restartsmemory_limiter first, queue sizes, resource limits
Partial traces after samplingOne trace must reach one Collector (load-balancing tier)
Some data missing intermittentlyQueue full or retry limit reached; check send_failed and queue metrics

Fix one hop at a time, keep the debug exporter handy while you work, and put the Collector’s internal metrics on a dashboard so the next incident starts with evidence.

Popular Guides