Keeping every trace gets expensive quickly, and most of them are routine requests that nobody will ever open. Sampling keeps a representative or interesting subset. OpenTelemetry offers two places to decide: in the SDK when a trace starts (head sampling) and in the Collector after the trace is complete (tail sampling). Both are pure configuration. This guide shows each and how to choose.

Head sampling in the SDK

The decision is made when the root span is created, so it is cheap and needs no extra infrastructure. It cannot know whether the request will fail or be slow, because it hasn’t happened yet.

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

The sampler names defined by the specification include always_on, always_off, traceidratio, parentbased_always_on (the default), parentbased_always_off, and parentbased_traceidratio. Use the parentbased_ variants: they follow the sampling decision of the incoming request’s parent, so a trace is either kept or dropped as a whole across services. A plain traceidratio in each service would make independent decisions and produce broken traces.

In the declarative configuration file, the same idea looks like:

tracer_provider:
  sampler:
    parent_based:
      root:
        trace_id_ratio_based:
          ratio: 0.1

Check your SDK’s documentation for the sampler names it supports in the file format.

Tail sampling in the Collector

Tail sampling waits until a trace is complete, then applies policies. It lets you keep every error and every slow trace plus a small share of the rest, which head sampling can’t do. The cost is more infrastructure and memory.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 100
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

How it works:

• decision_wait is how long the processor waits after seeing the first span of a trace before deciding. It must be longer than your slowest trace’s duration, or you will decide on partial traces.
• num_traces is how many traces are held in memory while waiting. Size it to roughly your trace arrival rate times decision_wait, with headroom.
• policies are evaluated for each trace. A trace is kept if any policy says to keep it, so the three above mean “keep errors, keep slow ones, and also keep 5 percent of everything”.
• Other policy types include string_attribute (match a value such as a customer tier), numeric_attribute, always_sample, and (combine policies), and composite (rate-allocate across policies).

tail_sampling is in the contrib distribution, and its option names evolve, so check its README for your version.

The catch: one trace must reach one Collector

Tail sampling only works if all spans of a trace arrive at the same Collector instance. With several replicas behind a plain load balancer, spans of one trace scatter and decisions are made on fragments. The standard solution is two tiers:

1. Tier 1 (load-balancing layer) uses the loadbalancing exporter to route spans by trace ID to a consistent tier-2 instance.
2. Tier 2 (sampling layer) runs tail_sampling and exports the kept traces.

Tier 1:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [loadbalancing]

Tier 2:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch:

exporters:
  otlp/backend:
    endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/backend]

Use a headless Service so that DNS lists the tier-2 pods, and note that when pods scale up or down, routing shifts and a few in-flight traces may be decided on incomplete data. Keeping the layers separate also isolates failures: a problem in sampling doesn’t block ingestion.

Combining head and tail

A common and sensible setup:

• A modest head sampler in the SDK (for example 50 percent) to cut volume at the source for very high-traffic services.
• Tail sampling in the Collector to guarantee that errors and slow requests survive.

Be careful: anything dropped at the head can’t be recovered by the tail. If you must keep all errors, don’t head-sample those services aggressively.

Metrics survive sampling if you derive them first

Dropping traces also drops the numbers you could compute from them, such as request rates and latency percentiles. Generate metrics before the sampling step with the spanmetrics connector, so dashboards stay accurate:

connectors:
  spanmetrics:

service:
  pipelines:
    traces/all:
      receivers: [otlp]
      exporters: [spanmetrics, loadbalancing]
    metrics/derived:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [otlphttp/prometheus]

Here every span feeds the connector, while only the sampled set continues to storage.

Choosing

Goal	Approach
Cut cost simply, no new infrastructure	Head sampling with parentbased_traceidratio
Keep all errors and slow requests	Tail sampling in a two-tier Collector
Keep accurate rates and percentiles	spanmetrics before sampling
Fewer services, low volume	No sampling; revisit when cost matters

Verify

• In tier 2, compare span counts in and out using the Collector’s own metrics (port 8888) to confirm the ratio matches your policies.
• Search for a known failed request and make sure its whole trace is present.
• Watch memory on tier 2, because it holds traces for decision_wait. If it grows, lower num_traces or add replicas, and keep memory_limiter in front.