Collecting and parsing logs is only half the pipeline; getting them somewhere useful, reliably, even when that destination is temporarily unreachable, is the other half. This guide covers installing an output plugin, routing multiple sources to their own destinations, and the buffering configuration that actually makes delivery reliable rather than best-effort.

Installing an output plugin

Fluentd’s built-in outputs cover the basics (file, stdout, forward to another Fluentd instance), but most real destinations, Elasticsearch, cloud storage, a SaaS log platform, come as separate plugins you install with fluent-gem, the Fluentd-managed wrapper around RubyGems:

sudo fluent-gem install fluent-plugin-elasticsearch

Using fluent-gem rather than a bare gem install matters: it installs into the Ruby environment Fluentd itself actually uses, rather than whatever the system’s default Ruby happens to be, which is a common source of “I installed the plugin but Fluentd still says it’s missing” confusion. If you’re running Fluentd as a container, install the plugin as part of building your own image layered on the official one, rather than installing it into a running container by hand, so it’s actually present the next time that container starts.

Configuring the output

Once installed, an Elasticsearch destination is just another <match> block:

<match apache.access>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name fluentd-apache
</match>

Routing different sources to different destinations

Nothing requires every source to end up in the same place. Multiple <match> blocks, each scoped to a different tag, send different data to different destinations from within the same configuration:

<match apache.access>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name fluentd-apache
</match>

<match docker.**>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name fluentd-docker
</match>

<match app.**>
  @type forward
  <server>
    host aggregator.internal
    port 24224
  </server>
</match>

Application logs going to one index, container logs to another, and a third source forwarded to an entirely separate Fluentd instance for further processing, all in the same pipeline. This is also how a single source can be sent to more than one destination at once: define two separate <match> blocks matching the same tag (or use a <copy> section), since a single Fluentd pipeline isn’t limited to one output per source.

Buffering: what actually makes delivery reliable

By default, a plugin buffers in memory before writing to its destination, which is fast, but means a Fluentd crash or restart loses whatever hadn’t been flushed yet. For anything where losing in-flight log data matters, configure an explicit file buffer instead, so buffered entries are persisted to disk and survive a restart:

<match apache.access>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name fluentd-apache

  <buffer>
    @type file
    path /var/log/fluent/buffer/apache
    flush_interval 5s
    retry_max_times 10
    retry_wait 5s
  </buffer>
</match>

This is the behavior people often describe as “Fluentd doesn’t lose data if it restarts,” but it’s worth being precise about: it’s true because of a configured file buffer, not automatically. With a memory buffer, a crash mid-flush still loses whatever was queued. If reliable delivery matters for a given destination, configure the file buffer for it explicitly rather than assuming the default covers it.

Handling a destination that’s temporarily down

The same buffering configuration is what handles a downed backend gracefully. If Elasticsearch (or whatever the destination is) becomes unreachable, entries accumulate in the buffer, rather than being dropped, and Fluentd retries delivery on the schedule retry_wait and retry_max_times define, with the buffered data safely on disk in the meantime if you’ve configured a file buffer. Once the destination is reachable again, Fluentd catches up from where it left off. This is what makes a brief network blip or a backend restart a non-event instead of a gap in your logs.

Redacting and enriching before the final output

A <filter> still runs before matching output, wherever it sits between source and match, so sensitive data can be stripped immediately before it reaches the destination:

<filter apache.access>
  @type record_transformer
  remove_keys remote_addr, user_agent
</filter>

record_transformer is the same plugin used for enrichment (see the previous guide’s hostname example); remove_keys is just the other direction, dropping fields rather than adding them. This is a reasonable place to redact anything sensitive, an email address, a session token, an internal IP, before it leaves your own infrastructure for a third-party destination, rather than relying on the destination to handle it correctly afterward.