Troubleshoot the Collector and SDKs
A systematic checklist for when telemetry doesn't arrive: debug exporter, health checks, internal metrics, and common mistakes.
“I don’t see my data” is the most common OpenTelemetry problem, and the cause is almost always one of a short list of configuration mistakes. Work from the application outward, one hop at a time, and prove each hop before moving to the next.
App SDK ──▶ Collector receiver ──▶ processors ──▶ exporter ──▶ Backend
1 2 3 4 5
Step 0: validate the config
otelcol-contrib validate --config=config.yaml
This catches YAML errors, unknown component names, bad option names, and pipelines that reference undefined components. Use the binary name that matches your distribution (otelcol, otelcol-contrib, otelcol-k8s). In a container, run it with the same image and mounted config as your deployment.
Step 1: is the SDK sending?
Enable SDK diagnostics and a console exporter to separate “the app produced nothing” from “the app produced it but it didn’t arrive”:
export OTEL_LOG_LEVEL=debug
export OTEL_TRACES_EXPORTER=console # or "logging", depending on the language
Check the usual causes:
- Endpoint and protocol don’t match.
OTEL_EXPORTER_OTLP_PROTOCOLmust match the port:grpcgoes to 4317,http/protobufgoes to 4318. Sending HTTP to the gRPC port, or the reverse, fails with errors that look like connection or parsing problems. - Wrong path with HTTP.
OTEL_EXPORTER_OTLP_ENDPOINTis a base URL; the SDK adds/v1/traces. Signal-specific variables such asOTEL_EXPORTER_OTLP_TRACES_ENDPOINTare used as given, so they must include the full path. localhostinside a container. The app’slocalhostis the app itself. Use the Collector’s Service or container name.- The SDK is disabled or the exporter is
none. Look forOTEL_SDK_DISABLED=trueorOTEL_TRACES_EXPORTER=none. - No
service.name. Data is accepted but appears asunknown_service, which looks like missing data. SetOTEL_SERVICE_NAME. - Short-lived processes exit before the export. Batch exporters send periodically, so call the SDK’s shutdown or flush before exit.
Step 2: is the receiver reachable?
curl -i http://collector:4318/v1/traces \
-H 'Content-Type: application/json' -d '{}'
An HTTP 200 response (an empty JSON body is fine) shows the HTTP receiver is reachable. For gRPC, a tool such as grpcurl or a TCP check (nc -zv collector 4317) proves the port is open. If it isn’t:
- The receiver may be bound to
localhost. Setendpoint: 0.0.0.0:4317explicitly. - The Kubernetes Service may not expose the port, or the pod’s
containerPortis missing. - A NetworkPolicy, firewall, or service mesh may block the traffic.
Step 3: is the pipeline wired up?
This is the most common Collector mistake: a component defined but not used. Defining exporters.otlp/tempo does nothing until it appears in service.pipelines.<signal>.exporters. The same goes for receivers, processors, extensions, and connectors.
Check:
- The component’s name in the pipeline matches its definition exactly, including any suffix (
otlp/tempo, nototlp). - The signal type matches: a
tracespipeline can’t use a receiver or exporter that doesn’t support traces. - The extension is listed under
service.extensions. - In Helm values, remember that lists replace, so a pipeline you partially redefined may have lost the default components.
Step 4: see the data inside the Collector
Add the debug exporter to the pipeline:
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug, otlp/tempo]
Read the Collector’s logs. Seeing spans there proves receivers and processors work, so the problem is downstream. Seeing nothing means data never reached the pipeline or a processor dropped it (a filter processor or the tail sampler is the usual suspect). Remove debug or lower its verbosity when finished, because it logs every item.
Step 5: use the Collector’s own signals
Health check:
extensions:
health_check:
endpoint: 0.0.0.0:13133
service:
extensions: [health_check]
curl http://collector:13133/ returns a status. Use it for Kubernetes liveness and readiness probes.
Internal metrics are served on port 8888 by default (check service.telemetry.metrics for your version):
curl -s http://collector:8888/metrics | grep -E 'otelcol_(receiver|processor|exporter)_'
Read them as a funnel:
| Metric family | Meaning |
|---|---|
otelcol_receiver_accepted_* | Items received successfully |
otelcol_receiver_refused_* | Rejected, often by memory_limiter |
otelcol_processor_dropped_* | Dropped by a processor |
otelcol_exporter_sent_* | Delivered to the backend |
otelcol_exporter_send_failed_* | Backend rejected or unreachable |
otelcol_exporter_queue_size and _capacity | Queue depth against its limit |
Accepted counts rising with sent counts flat tell you the problem is at the exporter. Refused counts point at memory limits. Metric names have been adjusted between releases, so check the Collector’s internal telemetry documentation for yours.
zPages give a live view of pipelines and recent traces:
extensions:
zpages:
endpoint: localhost:55679
Browse to /debug/tracez and /debug/pipelinez (through kubectl port-forward on a cluster). Keep it bound to localhost.
Step 6: is the backend accepting it?
Look at exporter errors in the Collector logs:
connection refusedordeadline exceeded: wrong endpoint, port, or a network block.certificate signed by unknown authority: addtls.ca_file, or fix the endpoint to use the right host name.401or403: the token or header is missing or wrong. Confirm the environment variable is set in the pod (kubectl exec ... -- env), because an unset${env:VAR}becomes an empty value.415or parsing errors: protocol mismatch, such asotlp(gRPC) pointed at an HTTP port.404on HTTP: check the base path the backend expects (Loki uses/otlp, Prometheus/api/v1/otlp).- Prometheus rejecting OTLP: its OTLP receiver feature flag isn’t enabled.
Auto-instrumentation checks
For Operator-based injection, confirm that the Instrumentation existed before the pod started, that the annotation is on the pod template, and that the application pod has the init container opentelemetry-auto-instrumentation. Restart the workload after creating or changing the resource. For agent-based setups, confirm the agent flag or environment variable is actually applied (-javaagent:, opentelemetry-instrument, --require).
Quick reference
| Symptom | First thing to check |
|---|---|
| Nothing in the Collector logs | SDK endpoint, protocol, port; localhost in containers |
| Collector receives data, backend doesn’t | Exporter in the pipeline; endpoint; auth; protocol |
| Collector fails to start | validate output; component missing from the image |
Data appears as unknown_service | Set OTEL_SERVICE_NAME |
| Memory growth or restarts | memory_limiter first, queue sizes, resource limits |
| Partial traces after sampling | One trace must reach one Collector (load-balancing tier) |
| Some data missing intermittently | Queue full or retry limit reached; check send_failed and queue metrics |
Fix one hop at a time, keep the debug exporter handy while you work, and put the Collector’s internal metrics on a dashboard so the next incident starts with evidence.
Popular Guides
-
OpenTelemetry Explained: SDKs, OTLP, and the Collector Understand the moving parts and the three places configuration lives before you write any YAML. -
The Collector Configuration Model: Components, Pipelines, and Providers Learn the rules of the Collector's YAML: naming, pipelines, connectors, merging files, and environment variables. -
Configure SDKs with OTEL_* Environment Variables Set the service name, exporters, sampling, and limits for any language using standard variables. -
Instrument Apps Without Code Changes: Java, Python, and Node.js Attach OpenTelemetry agents with environment variables and command-line flags only. -
Make the Collector Production-Ready: Memory Limits, Batching, and Retries Configure processors, queues, and persistent storage so telemetry survives overload and outages. -
Deploy the Collector on Kubernetes with Helm Install the official chart, use presets for logs and metrics, and override the config through values files.