The Collector is a single binary, but you can run it in several shapes. Which one fits depends on how many services you have, how much processing you need, and who owns the infrastructure. This guide describes the patterns and shows the configuration for the most common one: node-level agents forwarding to a central gateway.

The patterns

No Collector. Applications export OTLP straight to a backend. It is the simplest setup, with nothing extra to run, and fine for a prototype. The downsides: every application needs the backend address and credentials, changing destination means redeploying apps, and there’s nowhere central to filter, redact, or sample.

Agent. A Collector runs next to the workload: as a sidecar, or as one per node (a DaemonSet on Kubernetes). Apps send to localhost or the node, so export is fast and the app’s network path is short. Agents are good for host and node metrics, log collection from local files, and enriching data with local metadata. The downside is many instances to configure and upgrade.

Gateway. One or more Collectors run as a central service that applications or agents send to. A gateway is the place for heavy or global work: tail sampling, routing to several backends, rate limiting, and holding the credentials for the backends. It is a service you can scale horizontally behind a load balancer.

Agent plus gateway. The common production design: agents do local collection and light enrichment, and forward over OTLP to the gateway, which does the central processing and exports to backends. It keeps application config trivial (send to the node agent), keeps backend credentials in one place, and gives each layer one job.

App ─OTLP─▶ Agent (per node) ─OTLP─▶ Gateway (Deployment) ─▶ Backends
                │ host metrics, logs,            │ tail sampling, routing,
                │ k8s metadata                   │ credentials, retries

The agent configuration

An agent keeps processing minimal and forwards everything to the gateway:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      filesystem:
      network:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
  k8s_attributes:
    passthrough: true     # only record the pod IP; the gateway looks up the rest
  batch:

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc.cluster.local:4317
    tls:
      insecure: true      # use real TLS between layers in production
    sending_queue:
      enabled: true
      queue_size: 2000
    retry_on_failure:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, batch]
      exporters: [otlp/gateway]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [memory_limiter, k8s_attributes, batch]
      exporters: [otlp/gateway]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, batch]
      exporters: [otlp/gateway]

passthrough: true is the key for the two-layer design. The agent only adds the pod IP to each resource, without calling the Kubernetes API. The gateway can’t see where telemetry originated once it arrives from an agent, so it uses that IP to enrich.

The gateway configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  k8s_attributes:
    auth_type: serviceAccount
    passthrough: false
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
        - k8s.node.name
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: connection
  batch:

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlphttp/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, batch]
      exporters: [otlphttp/prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, batch]
      exporters: [otlphttp/loki]

The gateway reads the pod IP the agent attached, asks the Kubernetes API for that pod’s metadata, and adds namespace, pod, and deployment names. Backend endpoints differ by product, so confirm the exact OTLP paths in your backend’s documentation (the Grafana guides in this library cover the Grafana stack).

Kubernetes permissions for k8s_attributes

The processor reads pod metadata, so its service account needs read access:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-k8s-attributes
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "watch"]

Bind it to the service account that runs the processor. The Helm chart’s kubernetesAttributes preset creates the needed RBAC for you. When agents run as a DaemonSet and do their own lookups instead of passing through, set filter.node_from_env_var to a variable carrying the node name (from the downward API), so each agent only watches pods on its own node, which keeps API load low on big clusters.

Choosing a pattern

Situation	Use
Prototype, one or two services	No Collector, or one gateway
Need node metrics or log files	Agent as a DaemonSet
Several teams, central policy, tail sampling, backend credentials in one place	Agent plus gateway
Strict isolation per team	A gateway per team, optionally one shared platform gateway behind them

Operational notes

• Run at least two gateway replicas behind a Service, and give agents a queue and retries so a gateway restart doesn’t lose data.
• Use TLS between agents and the gateway outside a trusted network, and authenticate senders to the gateway where it is exposed.
• Pin the same Collector version across layers where you can, and upgrade the gateway first.
• If you add tail sampling, add the load-balancing tier from the sampling guide between agents and the sampler.