A Collector that works on a laptop can fall over in production: a traffic spike exhausts memory, a backend outage drops data, or a restart loses everything in flight. The fixes are all configuration. This guide covers the processors and exporter settings that every production Collector should have, and the order to put them in.

The baseline pipeline

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    timeout: 5s
    send_batch_size: 8192

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]

Order matters: processors run in the order listed. memory_limiter goes first so that backpressure reaches the receivers before anything else consumes memory. batch goes last, after any filtering or enrichment, so batches are built from the final data.

memory_limiter

It watches the Collector’s memory and refuses new data when usage gets too high, so the process slows down instead of being killed by the operating system.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
    spike_limit_mib: 800

• check_interval is how often memory is measured. A value of 1s is recommended, and the default of 0s disables the periodic check.
• limit_mib is the hard limit for heap memory. The whole process typically uses somewhat more.
• spike_limit_mib is the expected maximum jump between checks, and defaults to 20 percent of the limit. The soft limit is limit_mib minus spike_limit_mib (here 3200 MiB).
• limit_percentage and spike_limit_percentage express the same limits as a share of available memory (Linux with cgroups). They suit containers where the memory limit is set by Kubernetes. If limit_mib is also set, it takes precedence.

Above the soft limit, the processor refuses data by returning an error to the previous component, and well-behaved senders retry or slow down. Above the hard limit it also forces garbage collection. Set it a little below the container’s memory limit so it acts before the kernel does.

batch

Batching groups items so that exports are fewer and larger, which saves network and backend work.

processors:
  batch:
    timeout: 5s            # send after this long even if the batch isn't full
    send_batch_size: 8192  # target batch size
    send_batch_max_size: 0 # 0 means no upper bound

Tune timeout for latency and send_batch_size for throughput. Recent Collector versions can also batch inside an exporter’s queue; check the exporter helper documentation for your version.

Exporter queue and retries

Exporters that send over the network have a sending queue and a retry policy. They sit in the exporter, not in a processor, and they are what protects you from short backend outages:

exporters:
  otlp/backend:
    endpoint: backend.example.com:4317
    sending_queue:
      enabled: true
      queue_size: 5000        # default is 1000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m   # default is 300s

How to think about the numbers:

• The queue buffers batches while the backend is slow or down. A bigger queue rides out longer incidents but uses more memory.
• retry_on_failure retries with increasing delays, up to max_elapsed_time. After that the data is dropped.
• Size both for your volume and the longest outage you want to survive without loss.

Persistent queue: survive restarts

By default the queue lives in memory, so a crash loses what was queued. Back it with disk using the file_storage extension:

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage

exporters:
  otlp/backend:
    endpoint: backend.example.com:4317
    sending_queue:
      storage: file_storage

service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]

Three pieces are required: define the extension, reference it in the exporter, and enable it under service.extensions. Mount a persistent volume at that directory, for example a PersistentVolumeClaim for a StatefulSet. Data can still be lost if the disk fills or fails, or if the backend stays down past the retry limit, so monitor disk usage.

Useful processors beyond the basics

Add or change attributes:

processors:
  resource/env:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
  attributes/scrub:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: hash

resource edits attributes that describe the producer (applied to a whole batch), and attributes edits per-span, per-metric-point, or per-log-record attributes. Use them to add environment labels and to remove or hash sensitive values before data leaves your network.

Drop noise with the filter processor, written in OpenTelemetry Transformation Language (OTTL):

processors:
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/healthz"'
        - 'attributes["url.path"] == "/ready"'

Transform data with the transform processor:

processors:
  transform/normalize:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - set(attributes["team"], "payments") where resource.attributes["service.name"] == "checkout"

OTTL is powerful, so test statements with the debug exporter on a copy of your data before enabling them in production. The statement syntax and available contexts change between versions, so check the processor’s README for yours. These processors are in the contrib distribution.

Protect the Collector itself

• Set Kubernetes resource requests and limits, and keep memory_limiter below the limit.
• Add health_check and use it for liveness and readiness probes.
• Monitor the Collector’s own metrics on port 8888 for queue size, dropped items, and refused data. Alert when otelcol_exporter_queue_size approaches its capacity or when exporters report send failures.
• Run more than one replica behind a Service for gateways, and size a single replica for the burst you expect.

Checklist

• memory_limiter is the first processor in every pipeline.
• batch is the last processor.
• Network exporters have a sending queue and retry policy sized for your outage tolerance.
• A persistent queue is configured where data loss is unacceptable.
• Sensitive attributes are deleted or hashed at the edge.
• Health checks, resource limits, and Collector metrics are in place.