Skip to content

Configuration

Vector reads one file, /etc/vector/vector.yaml. This page explains the default pipeline, the redaction step, and how to add an S3 sink.

Where things live

Path What it is
/etc/vector/vector.yaml The Vector configuration (root and vector only, mode 0640)
/var/lib/vector/vector.env The HTTP ingest password (root and vector only, mode 0640)
/etc/vector/tls/cert.pem This instance's certificate, readable by everyone for clients
/var/lib/vector/tls/key.pem Its private key (root and vector only, mode 0640)
/var/lib/vector/in/ File ingest: *.log files, one JSON object per line
/var/lib/vector/out/ Local destination: accepted-YYYY-MM-DD.log
/var/lib/vector/data/ Disk buffer
/opt/sushi/etc/vector/redact.vrl A reference copy of the redaction program
/opt/sushi/etc/vector/aws-s3-sink.yaml.example Example S3 sink, not loaded
/opt/sushi/etc/vector/iam-s3-sink.json.example Example IAM policy for that sink

First boot writes vector.yaml only if it does not exist yet, so your edits are kept across reboots.

The default pipeline

Sources

  • http_in: HTTPS on 127.0.0.1:8687, path /ingest, JSON, basic authentication with user vector. Acknowledgements are on.
  • file_in: reads /var/lib/vector/in/*.log from the beginning.
  • internal_metrics: Vector's own metrics.

Transform

  • redact: a VRL remap step on http_in and file_in. It parses a JSON line from the file source, drops records whose level is debug, replaces password, token and email with REDACTED, and adds pipeline: sushi-vector.

Sinks

  • local_file: newline-delimited JSON to /var/lib/vector/out/accepted-%Y-%m-%d.log, through a disk buffer of 268435488 bytes that blocks when full, so back-pressure reaches the sources.
  • metrics: Prometheus exporter on 127.0.0.1:9598.

The API listens on 127.0.0.1:8686.

Redaction is not a complete filter

The redaction step only handles the three field names above. Other sensitive values pass through unchanged. Extend the VRL program for your own data before you send logs to a remote sink.

Apply changes

The service validates the configuration before it starts. To check your changes first:

sudo nano /etc/vector/vector.yaml
sudo -u vector /opt/vector/bin/vector --config /etc/vector/vector.yaml validate
sudo systemctl restart vector

Add an S3 sink

  1. Copy the example policy and edit the bucket name:

    cat /opt/sushi/etc/vector/iam-s3-sink.json.example

    It allows s3:ListBucket on the bucket, for the sink's health check, and s3:PutObject on vector/* in that bucket.

  2. Create an IAM role with that policy and attach it to the instance. Vector uses the standard AWS credential chain, so you do not need access keys in the config.

  3. Add the sink from /opt/sushi/etc/vector/aws-s3-sink.yaml.example under sinks: in /etc/vector/vector.yaml, and set your region and bucket. It takes its input from redact, writes gzip-compressed JSON lines under vector/date=%Y-%m-%d/, and has its own disk buffer.
  4. Validate and restart Vector, then check the log.

Change the ingest password

The password is stored in two places: /var/lib/vector/vector.env and, written in at first boot, the password field of http_in in /etc/vector/vector.yaml. To change it, update both files with the same new value, then restart Vector.

Accept logs from other hosts

Everything listens on 127.0.0.1. To accept logs from other hosts in your VPC, either leave Vector on loopback and put a proxy you control in front of it, or change the address of http_in to a private address and allow only your own network in the security group. Do not bind ingest, the API or the metrics exporter to 0.0.0.0 or open them to the internet.

The walkthrough checks the defaults

demo.sh and the self-test it runs check the image's default settings. After you change the configuration, they can report a failure even when your setup works. Run them on a fresh instance, before you make changes.

Next: Operations.