Troubleshooting¶
Fixes for the problems you are most likely to hit with the Fluent Bit Log Gateway.
curl to the ingest port fails¶
Cause: the request uses plain HTTP, or the client does not trust the instance's certificate.
Fix: use https:// and pass the public certificate:
--cacert /usr/local/share/sushi/fluent-bit/cert.pem. Plain HTTP to port 9880
is refused by design.
Records do not appear in accepted.log¶
Cause: the record was not valid JSON, its tag does not match an output, or the output is failing.
Fix: check that the request returned HTTP 201, then read the Fluent Bit log:
Fluent Bit does not start after a config change¶
Cause: usually an error in /etc/fluent-bit/fluent-bit.conf.
Fix: read the error in sudo journalctl -u fluent-bit -n 50 --no-pager,
fix the file and restart with sudo systemctl restart fluent-bit.
A CloudWatch Logs or S3 output fails¶
Cause: usually a missing IAM role or permission, a wrong region, or a log group or bucket that does not exist.
Fix: check that the instance has a role attached with the right permissions for that resource, that the names and region in the config are correct, and read the Fluent Bit log for the AWS error.
The disk is filling up¶
Cause: accepted.log is not rotated.
Fix: see Disk use. Check which directory is growing, and set up your own retention for the local destination, or move delivery to CloudWatch Logs or S3.
First boot failed¶
Cause: sushi-fluent-bit-firstboot.service failed, so Fluent Bit did
not start.
Fix: look at its log, then run it again and start Fluent Bit:
sudo journalctl -u sushi-fluent-bit-firstboot.service --no-pager
sudo systemctl restart sushi-fluent-bit-firstboot
sudo systemctl start fluent-bit
If it still fails, email info@softwaresushi.com with the instance ID and the log output.