Configuration

Fluent Bit reads one file, /etc/fluent-bit/fluent-bit.conf. This page explains the default pipeline and how to add your own outputs.

Where things live

Path	What it is
/etc/fluent-bit/fluent-bit.conf	The Fluent Bit configuration
/var/lib/fluent-bit/storage/	Filesystem buffer
/var/lib/fluent-bit/out/accepted.log	The default local destination
/var/lib/fluent-bit/tls/	This instance's certificate and private key
/usr/local/share/sushi/fluent-bit/cert.pem	Public copy of the certificate, for clients
/opt/fluent-bit/bin/fluent-bit	The Fluent Bit binary

First boot writes fluent-bit.conf only if it does not exist yet, so your edits are kept across reboots.

The default pipeline

• Service. The monitoring API listens on 127.0.0.1:2020. The buffer is in /var/lib/fluent-bit/storage.
• Input. An http input named local-ingest listens on 127.0.0.1:9880 with TLS, tags records ingest.local, returns HTTP 201 on success, and exposes /health. It uses the filesystem buffer, with Mem_Buf_Limit 2MB.
• Output. A file output named local-file matches ingest.local and writes plain lines to /var/lib/fluent-bit/out/accepted.log. Its queued chunks are limited by storage.total_limit_size 32M, and it retries a failed write up to five times (Retry_Limit 5).

Apply changes

sudo nano /etc/fluent-bit/fluent-bit.conf
sudo systemctl restart fluent-bit
systemctl is-active fluent-bit

If the service does not come back, the log shows why:

sudo journalctl -u fluent-bit -n 50 --no-pager

Send logs to CloudWatch Logs

1. Create an IAM role that allows Fluent Bit to write to your log group, scoped to that log group, and attach it to the instance. Do not put access keys in the config file.

2. Add an output to fluent-bit.conf:

   [OUTPUT]
       Name                  cloudwatch_logs
       Match                 ingest.local
       region                <region>
       log_group_name        <your-log-group>
       log_stream_prefix     fluent-bit-
       auto_create_group     Off
       storage.total_limit_size 32M
       Retry_Limit           5
   

3. Restart Fluent Bit and check the log.

Send logs to S3

1. Create an IAM role that allows Fluent Bit to write to your bucket, scoped to that bucket, and attach it to the instance.

2. Create a local store directory for the S3 output. Keep it under /var/lib/fluent-bit, because that is the only place the service can write:

   sudo install -d -m 0750 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/s3
   

3. Add an output:

   [OUTPUT]
       Name                  s3
       Match                 ingest.local
       bucket                <your-log-bucket>
       region                <region>
       total_file_size       1M
       upload_timeout        1m
       use_put_object        On
       store_dir             /var/lib/fluent-bit/s3
       store_dir_limit_size  32M
       preserve_data_ordering On
   

4. Restart Fluent Bit and check the log.

The S3 output has its own buffer: store_dir_limit_size limits it, not storage.total_limit_size. Include that directory in your backups. Test the output with your own bucket and role before you rely on it.

Accept logs from other hosts

Ingest listens on 127.0.0.1 only. To accept logs from other hosts in your VPC, either leave Fluent Bit on loopback and put a proxy you control in front of it, or change Listen in the [INPUT] section to a private address and allow only your own network in the security group.

Ingest has no client authentication

The certificate lets clients verify the server, but the input does not ask clients for a certificate or token. Any process that can reach the port can send logs. Do not bind ingest or the monitoring API to 0.0.0.0 or open them to the internet.

The walkthrough checks the defaults

demo.sh and the self-test it runs check the image's default settings. After you change the configuration, they can report a failure even when your setup works. Run them on a fresh instance, before you make changes.

Next: Operations.