A Fluentd pipeline, collect, transform, ship, is expressed entirely through one configuration file, built from a small set of directives. This guide covers what each directive does, how tags connect them together, and how to check a configuration before trusting it with real traffic.

Where the config file lives

Fluentd’s own packaged distribution is called fluent-package (the successor to the older td-agent name; if you’re reading documentation or tutorials that mention td-agent, it’s the same project under its previous branding). Installed this way, the configuration file is typically at /etc/fluent/fluentd.conf. A separate distribution, Calyptia Fluentd, maintained by a different organization, uses its own path. Check your installation method’s documentation for the exact location rather than assuming one.

The three core directives

• <source> defines where log data comes from. Every source declares a plugin @type (tail to follow a file, http to accept entries over HTTP, and others) and, critically, a tag, the label every entry from this source carries through the rest of the pipeline.
• <filter> processes entries that match a given tag pattern, in place, before they reach an output. Common uses: adding fields (enrichment), removing fields (redaction), or dropping entries that match a pattern entirely.
• <match> is where matching entries are actually sent. Like <filter>, it’s scoped to a tag pattern, but instead of transforming data in place, it’s the pipeline’s output stage: a destination and the plugin @type that knows how to write to it.

<source>
  @type tail
  path /var/log/apache2/access.log
  pos_file /var/log/fluent/apache-access.log.pos
  tag apache.access
  <parse>
    @type apache2
  </parse>
</source>

<filter apache.access>
  @type record_transformer
  remove_keys host
</filter>

<match apache.access>
  @type elasticsearch
  host localhost
  port 9200
  index_name fluentd-apache
</match>

Reading this top to bottom: the source tails Apache’s access log, parses each line using Fluentd’s built-in Apache log format parser, and tags every resulting entry apache.access. The filter, scoped to that same tag, strips the host field from each entry before it goes further. The match, also scoped to that tag, ships the (now redacted) entries to Elasticsearch.

Tag matching, not just tag equality

A <filter> or <match> block’s tag pattern doesn’t have to be an exact string. apache.* matches any tag starting with apache.; ** matches everything. This is what makes routing flexible: one <match **> block can act as a catch-all destination, while more specific patterns handle particular sources differently, redacting one source’s data but not another’s, or shipping different sources to different destinations entirely.

Reading data: the nested <parse> block

Turning a raw log line into structured key-value data is the job of a parser, configured as a nested <parse> block inside a source (this nested form is current Fluentd syntax; older configuration examples sometimes show parsing options as flat parameters directly on the source, which is outdated style and no longer how current versions expect it written). Fluentd ships with parsers for common formats out of the box, json, apache2, nginx, syslog, csv, and a regexp parser for anything that doesn’t match a built-in format:

<parse>
  @type json
</parse>

Picking the right parser for a source is what turns an unstructured text line into fields you can filter, enrich, and query individually downstream, rather than one opaque blob of text.

Structuring a config with includes

Once a configuration covers several sources, keeping everything in one file gets unwieldy. The @include directive splits configuration across multiple files:

# fluentd.conf
@include apache.conf
@include containers.conf
@include outputs.conf

Each included file holds its own <source>, <filter>, or <match> blocks, and Fluentd assembles them as if they were written inline. This is purely organizational; it doesn’t change how tags, matching, or processing work, just how readable the configuration stays as it grows.

Validate before you run

Before starting Fluentd for real, check that a configuration is actually well-formed:

fluentd --dry-run -c /etc/fluent/fluentd.conf

This parses and validates the configuration without processing any actual log data. A clean dry run doesn’t guarantee the pipeline does what you intend, that still needs testing with real data, but it does catch outright syntax errors and plugin misconfiguration before they surface as a confusing runtime failure.

Run it and watch it work

For manual testing (as opposed to running Fluentd as a managed background service, which is how it normally runs in production):

fluentd -c /etc/fluent/fluentd.conf

With a source tailing a file and a match writing to @type stdout, you can watch entries flow through the pipeline in real time as new lines are written to the source file, which is a fast way to confirm a new source or filter is actually doing what you expect before pointing it at a real destination.