Most real deployments need more than one kind of input source at once: an existing application writing to a log file, a service that can push logs over HTTP instead, and every container running on a host. This guide configures all three, building on the source/filter/match structure from the previous guide.

Tailing a log file

The tail input plugin follows a file the way tail -f does, picking up new lines as they’re written:

<source>
  @type tail
  path /var/log/apache2/access.log
  pos_file /var/log/fluent/apache-access.log.pos
  tag apache.access
  <parse>
    @type apache2
  </parse>
</source>

pos_file is worth taking seriously even though it’s easy to skip: it’s where Fluentd records exactly how far into the file it has already read. Without it, restarting Fluentd means re-reading the entire file from the start, duplicating everything already shipped. With it, a restart picks up exactly where it left off. The plugin’s own documentation describes it as highly recommended for precisely this reason.

Accepting logs over HTTP

Not everything can have Fluentd installed alongside it. A managed cloud service, a serverless function, anything outside infrastructure you control at the OS level, can’t run a tail source next to a file it’s writing. For those, the http input plugin lets something send logs to Fluentd instead of Fluentd reading from it:

<source>
  @type http
  port 9880
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

Anything that can make an HTTP POST request, a custom application, a logging library configured to forward there, a curl call in a script, can now deliver log data directly. body_size_limit and keepalive_timeout are worth setting deliberately rather than leaving at whatever default your version ships with, since an HTTP source is exposed to whatever traffic reaches it, and unbounded request sizes or held-open connections are the kind of thing you want under your own control rather than default behavior.

Collecting every container’s logs on a host

Docker (and container runtimes generally) write each container’s stdout/stderr to JSON-formatted log files on the host, one directory per container. Pointing a tail source at all of them with a wildcard is enough to collect every container’s output without configuring each one individually:

<source>
  @type tail
  path /fluentd/log/containers/*/*-json.log
  pos_file /var/log/fluent/docker.log.pos
  tag docker.*
  read_from_head true
  <parse>
    @type json
  </parse>
</source>

For this to work, the host’s Docker log directory (/var/lib/docker/containers by default) needs to actually be mounted into wherever Fluentd is running, at whatever path the source’s path pattern points to. This is precisely how Fluentd works as a Kubernetes DaemonSet too: one Fluentd Pod per node, each mounting that node’s own container log directory, so across a whole cluster, every node’s containers get collected the same way without per-container configuration.

read_from_head true tells Fluentd to start from the beginning of files it hasn’t seen yet, which matters for containers that were already running (and already logging) before this source was configured; without it, Fluentd only picks up new lines from the moment it starts watching a given file.

Enriching with context: which container, which host

Raw container log lines don’t say which pod, node, or container they came from on their own; that context has to be attached, usually with a filter using the record_transformer plugin:

<filter docker.**>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

This attaches the current host’s name to every entry tagged under docker., which is what lets you later tell which of several replicas, behind a load balancer or across a cluster, a given log line actually came from. The same pattern extends to pod name, namespace, or any other metadata worth attaching at collection time rather than trying to reconstruct later.

Verify each source independently

Before wiring up real output destinations, point each source at a simple @type stdout match and confirm it’s producing the structure you expect:

<match apache.access>
  @type stdout
</match>

Generate a request (or write a line to the file, or POST to the HTTP endpoint, or start a container) and watch the parsed, tagged output appear. Confirming each source in isolation, before adding filters or real destinations on top, makes it much easier to tell which stage introduced a problem if something later in the pipeline doesn’t look right.