LiteLLM Proxy AI Platform & Data · LiteLLM 1.104.2 Launch on AWS

Configuration¶

This page shows where the gateway's configuration lives and how to make the common changes: add model providers, keep their API keys out of the config file, rotate the master key and reach the gateway from other machines. Make every change as root, then restart the proxy.

Where things live¶

Path	What it is
`/etc/litellm-proxy/config.yaml`	Models, routing and settings. Created on first boot and not overwritten afterwards.
`/etc/litellm-proxy/runtime.env`	Service environment: master key, host, port and TLS paths. Rewritten on every boot.
`/var/lib/litellm-proxy/master.key`	The master key. Root only.
`/var/lib/litellm-proxy/tls/`	`cert.pem` and `key.pem` for HTTPS.
`/opt/sushi/etc/litellm-proxy/config.yaml.tmpl`	The default configuration, used when `config.yaml` is missing.
`/opt/litellm-proxy/venv`	The LiteLLM installation.

Do not edit runtime.env

The first-boot service rewrites runtime.env from the master key on every boot, so anything you add there is lost. Put provider keys in a separate file, as shown below.

The default configuration¶

model_list:
  - model_name: smoke-mock
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: "sk-not-used"
      mock_response: "sushi-litellm-proxy-smoke-ok"

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

smoke-mock never calls a provider. Keep the master_key line so the gateway keeps requiring the key.

Add a provider¶

1. Store the provider key¶

Create a root-only environment file for your provider keys:

sudo install -m 0600 -o root -g root /dev/null /etc/litellm-proxy/providers.env
sudo nano /etc/litellm-proxy/providers.env

Add one NAME=value line per key, for example:

OPENAI_API_KEY=<your-openai-key>

Then tell the service to load it, with a systemd drop-in:

sudo mkdir -p /etc/systemd/system/litellm-proxy.service.d
printf '[Service]\nEnvironmentFile=/etc/litellm-proxy/providers.env\n' \
  | sudo tee /etc/systemd/system/litellm-proxy.service.d/providers.conf
sudo systemctl daemon-reload

systemd reads the file as root before it starts the proxy, so the file can stay root only.

2. Add the model¶

Add an entry to model_list in /etc/litellm-proxy/config.yaml that refers to the key by its variable name:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

model_name is the name your clients send. model is the provider and model, in LiteLLM's provider/model form. See the LiteLLM provider list for the exact settings each provider needs.

3. Restart and test¶

sudo systemctl restart litellm-proxy
sudo journalctl -u litellm-proxy -n 50 --no-pager

Then send a chat completion with your new model_name, as in Send a chat completion.

Amazon Bedrock¶

For Bedrock you do not need a key file. Attach an IAM role to the instance that allows bedrock:InvokeModel on the models you use, then add an entry such as:

  - model_name: claude
    litellm_params:
      model: bedrock/<bedrock-model-id>
      aws_region_name: <region>

Scope the role to the models and actions you need, and never put AWS access keys in config.yaml.

Rotate the master key¶

First boot does not replace an existing master key. To rotate, remove it and let the first-boot service create a new one. It also rewrites runtime.env:

sudo systemctl stop litellm-proxy
sudo rm /var/lib/litellm-proxy/master.key
sudo systemctl restart sushi-litellm-proxy-firstboot
sudo systemctl start litellm-proxy

Then update the key in every client. A new key must start with sk-.

Replace the TLS certificate¶

The certificate created on first boot is valid for 825 days. To create a new one, remove both files and let the first-boot service recreate them:

sudo systemctl stop litellm-proxy
sudo rm /var/lib/litellm-proxy/tls/cert.pem /var/lib/litellm-proxy/tls/key.pem
sudo systemctl restart sushi-litellm-proxy-firstboot
sudo systemctl start litellm-proxy

Give your clients the new cert.pem.

Reach the gateway from other machines¶

The proxy listens on 127.0.0.1:4000 only; the service passes that address on its command line. Choose one of these:

Session Manager port forwarding for your workstation, see Connect from your workstation.
Run your application on the same instance.
A reverse proxy on the instance that you install and manage, which forwards to https://127.0.0.1:4000. Bind it to the instance's private address, use a certificate your clients trust, and in the security group allow its port only from your applications' security groups.

Keep it private

Do not open port 4000 to the internet, and do not remove the master key. Anyone with the key can spend your provider credits.

Last updated October 13, 2026 · Suggest an improvement