The Collector Configuration Model: Components, Pipelines, and Providers
Learn the rules of the Collector's YAML: naming, pipelines, connectors, merging files, and environment variables.
Every Collector setup is one YAML document that follows the same few rules. Learning those rules once lets you read any Collector config, including the ones that ship with Helm charts and the Operator. This guide covers the structure, the naming scheme, how components are switched on, how to split and merge files, and how to inject environment variables.
The five component types
| Section | Job | Examples |
|---|---|---|
receivers | Take data in | otlp, prometheus, filelog, hostmetrics |
processors | Transform, filter, enrich, batch | batch, memory_limiter, k8s_attributes, tail_sampling |
exporters | Send data out | otlp, otlphttp, debug, prometheus |
connectors | Join two pipelines, acting as an exporter in one and a receiver in another | spanmetrics, forward, routing |
extensions | Add capabilities beside the pipelines | health_check, zpages, file_storage, auth extensions |
A sixth section, service, wires them together.
Components are off until service enables them
This is the single most common source of confusion. Configuring a receiver does not turn it on. A component only takes effect when a pipeline in service references it:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
debug:
otlp_grpc:
endpoint: gateway.example.com:4317
service:
extensions: [health_check, zpages]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp_grpc]
metrics:
receivers: [otlp]
exporters: [otlp_grpc]
logs:
receivers: [otlp]
exporters: [otlp_grpc]
A component that is defined but never referenced is ignored, which is useful for keeping variants in a file but easy to forget when nothing arrives. The pipelines are named traces, metrics, and logs, and a receiver, processor, or exporter must support the signal of the pipeline it joins.
Naming: type and type/name
A component identifier is its type, optionally followed by /name. Use names to configure the same component type more than once:
exporters:
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
otlp/vendor:
endpoint: otlp.vendor.example.com:4317
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp/tempo, otlp/vendor] # fan out to two backends
Each identifier must be unique. The fan-out above sends every span to both destinations.
Pipelines in detail
- Processor order matters. Data passes through processors in the order listed. Put
memory_limiterfirst andbatchlast (see the resiliency guide). - A component can appear in several pipelines. Each pipeline that uses a processor gets its own instance.
- Connectors bridge pipelines. A connector is listed as an exporter of one pipeline and a receiver of another. For example,
spanmetricsturns spans into metrics:
connectors:
spanmetrics:
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/tempo, spanmetrics]
metrics:
receivers: [otlp, spanmetrics]
processors: [batch]
exporters: [otlphttp/prometheus]
Environment variables
Use ${env:NAME} anywhere a value is expected, and bash-style defaults with :-:
exporters:
otlp:
endpoint: ${env:OTLP_ENDPOINT:-gateway.example.com:4317}
headers:
authorization: Bearer ${env:API_TOKEN}
To write a literal dollar sign, use $$. Because secrets arrive by environment variable, the file itself stays free of them and can be committed.
Config sources and merging files
--config accepts a path, or a scheme:value URI with the schemes file, env, yaml, http, and https:
otelcol --config=file:/etc/otelcol/base.yaml --config=file:/etc/otelcol/overrides.yaml
otelcol --config=env:MY_CONFIG_IN_AN_ENVVAR
otelcol --config="yaml:exporters::debug::verbosity: normal"
When you pass several --config options, the Collector merges them into one configuration. Each file may be partial, but the merged result must be complete, because the Collector doesn’t add components for you. Use this to keep a shared base file and a small per-environment override.
You can also embed another file into a value:
exporters: ${file:exporters.yaml}
The http and https providers load configuration from a URL, which is convenient but means the Collector’s behavior depends on that server, so use them deliberately.
Validate
otelcol validate --config=config.yaml
# or with the contrib image
docker run --rm -v $(pwd)/config.yaml:/etc/otelcol/config.yaml:ro \
otel/opentelemetry-collector-contrib:0.163.0 validate --config=/etc/otelcol/config.yaml
validate catches unknown components, bad syntax, and pipelines that reference something you didn’t define. It doesn’t prove that a backend is reachable.
Security default to know
Receivers bind to localhost by default. That’s safe when every sender runs on the same host, but it means a Collector in a container is unreachable from outside until you set an endpoint such as 0.0.0.0:4317. When you open an endpoint to the network, add TLS and authentication (see the backends guide), and restrict access with network policy or firewall rules.
A readable layout for real projects
collector/
├── base.yaml # receivers, processors, shared pipelines
├── env/
│ ├── dev.yaml # exporters and endpoints for dev
│ └── prod.yaml
└── README.md
Run it with --config=file:base.yaml --config=file:env/prod.yaml. Reviewers can see exactly what changes per environment.
Quick reference
- Define components at the top, enable them in
service. - Use
type/nameto repeat a component type. - Order processors deliberately.
- Use
${env:VAR}for anything environment-specific or secret. - Merge several
--configfiles, and runvalidatein CI.
Popular Guides
-
OpenTelemetry Explained: SDKs, OTLP, and the Collector Understand the moving parts and the three places configuration lives before you write any YAML. -
Configure SDKs with OTEL_* Environment Variables Set the service name, exporters, sampling, and limits for any language using standard variables. -
Instrument Apps Without Code Changes: Java, Python, and Node.js Attach OpenTelemetry agents with environment variables and command-line flags only. -
Make the Collector Production-Ready: Memory Limits, Batching, and Retries Configure processors, queues, and persistent storage so telemetry survives overload and outages. -
Deploy the Collector on Kubernetes with Helm Install the official chart, use presets for logs and metrics, and override the config through values files. -
Troubleshoot the Collector and SDKs A systematic checklist for when telemetry doesn't arrive: debug exporter, health checks, internal metrics, and common mistakes.