OpenTelemetry OpenTelemetry

Use the OpenTelemetry Operator for Collectors and Auto-Instrumentation

Declare Collectors and instrumentation as Kubernetes resources, and inject agents with pod annotations.

15 min Advanced

The OpenTelemetry Operator manages two things as Kubernetes resources: Collectors (an OpenTelemetryCollector object) and auto-instrumentation (an Instrumentation object plus pod annotations that tell the Operator to inject an agent). Application teams get telemetry by adding an annotation, not by editing Dockerfiles, and platform teams keep the settings in one place in Git.

Install the Operator

The Operator uses admission webhooks, which need TLS certificates. The simplest route is cert-manager, which the chart expects by default. Without cert-manager, let Helm generate a self-signed certificate:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install my-opentelemetry-operator open-telemetry/opentelemetry-operator \
  --namespace opentelemetry-operator-system --create-namespace \
  --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.autoGenerateCert.enabled=true

With cert-manager already installed, you can leave the certificate settings at their defaults, or reference your own issuer through admissionWebhooks.certManager.issuerRef. Pin the chart version with --version, and put the values in a file or a Flux HelmRelease rather than --set flags.

Declare a Collector

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: demo
  namespace: observability
spec:
  mode: deployment        # deployment, daemonset, statefulset, or sidecar
  replicas: 2
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 20
      batch: {}
    exporters:
      debug: {}
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug]

The config block is the same Collector YAML you’ve been writing, embedded in the resource. The Operator creates the workload and a Service named <name>-collector (here demo-collector) exposing the ports your receivers use. The available mode values map to the patterns in the deployment guide: daemonset for agents, deployment for gateways, and sidecar for a Collector injected into application pods. Check the CRD version available in your cluster (kubectl get crd opentelemetrycollectors.opentelemetry.io), since the API version has changed over releases.

Declare instrumentation settings

An Instrumentation resource holds the SDK settings that the Operator injects into annotated pods:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
  namespace: webapp
spec:
  exporter:
    endpoint: http://demo-collector.observability.svc.cluster.local:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  java:
    env:
      - name: OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED
        value: "true"

Important details:

  • exporter.endpoint defaults to http://localhost:4317 if omitted, which usually delivers nothing. Always set it to your Collector’s Service.
  • For a Collector in another namespace, use the full DNS name, <service>.<namespace>.svc.cluster.local.
  • The documented examples use port 4318 (HTTP) for .NET, Go, Java, and Python, and 4317 (gRPC) for Node.js. Match the port to the protocol the language agent uses.
  • Language-specific environment variables go under spec.<language>.env, such as spec.java.env or spec.python.env.

Create the Instrumentation before the application. The Operator injects the agent when a pod is created. If the resource doesn’t exist yet, the pod starts without the init container and you have to restart it.

Opt workloads in with annotations

Add an annotation to the pod template of a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: webapp
spec:
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
        - name: checkout
          image: myorg/checkout:1.4.2

The annotation per language:

LanguageAnnotation
Javainstrumentation.opentelemetry.io/inject-java: "true"
Pythoninstrumentation.opentelemetry.io/inject-python: "true"
Node.jsinstrumentation.opentelemetry.io/inject-nodejs: "true"
.NETinstrumentation.opentelemetry.io/inject-dotnet: "true"
Goinstrumentation.opentelemetry.io/inject-go: "true" (plus the target executable annotation)
SDK settings onlyinstrumentation.opentelemetry.io/inject-sdk: "true"

The value can be "true" (use the Instrumentation in this namespace), the name of one ("my-instrumentation"), a namespace-qualified name ("other-ns/my-instrumentation"), or "false". Put the annotation on a namespace to instrument every workload in it.

Related annotations you will use:

  • instrumentation.opentelemetry.io/container-names: "app,worker" chooses which containers to instrument. By default only the first container is.
  • resource.opentelemetry.io/service.name: checkout sets resource attributes such as the service name. Others include service.version and deployment.environment.name.
  • instrumentation.opentelemetry.io/inject-sdk: "true" injects only the environment variables, for apps that already include an SDK.
  • A single container can’t have more than one language injected. Multi-container, multi-language setups need the enable-multi-instrumentation feature flag and per-language container-name annotations.
  • Go requires the enable-go-instrumentation feature gate, the otel-go-auto-target-exe annotation (or the OTEL_GO_AUTO_TARGET_EXE setting), and a privileged sidecar, so review the security implications first.

Apply and verify

kubectl apply -f collector.yaml -f instrumentation.yaml
kubectl apply -f checkout-deployment.yaml

kubectl get opentelemetrycollectors,instrumentations -A
kubectl get pod -n webapp -l app=checkout -o jsonpath='{.items[0].spec.initContainers[*].name}'; echo
kubectl logs -n observability deploy/demo-collector | grep -i 'service.name'

An init container named opentelemetry-auto-instrumentation on the application pod means injection worked. Generate some traffic, and the Collector’s debug exporter should print spans with your service name.

Troubleshooting

  1. Is the Instrumentation resource there? kubectl describe otelinst -n <namespace>.
  2. Operator errors? Read the manager container’s logs in the Operator namespace.
  3. Was it created before the app? If not, restart the Deployment so the pod is recreated and injected. Check kubectl get events -n <namespace> for the init container.
  4. Is the annotation right? It must be on spec.template.metadata.annotations, and the language must match the app.
  5. Is the endpoint right? A wrong protocol or port, or a short Service name across namespaces, silently sends nothing.

When to prefer the Operator

Use it when many teams deploy to the cluster and you want telemetry to be opt-in by annotation, with settings owned centrally. Use the plain Helm chart when you only need a Collector and handle SDK configuration in each application’s own manifests. The Operator and the chart can coexist, but don’t manage the same Collector with both.

Popular Guides