Skip to content
LiteLLM Proxy AI Platform & Data · LiteLLM 1.104.2 Launch on AWS

Getting started

This page takes you from the AWS Marketplace listing to a working gateway: subscribe, launch the instance, connect, and send your first chat completion to the built-in mock model.

You need: an AWS account with permission to subscribe to AWS Marketplace products, launch EC2 instances and use AWS Systems Manager. To call the gateway from your workstation, you also need the AWS CLI with the Session Manager plugin.

1. Subscribe

Open the LiteLLM Proxy listing in AWS Marketplace, choose View purchase options, review the terms and subscribe. When the subscription is confirmed, choose Launch your software.

2. Launch the instance

On the EC2 launch page:

  1. Keep the instance type at t3.medium, or choose one with at least 4 GiB of memory.
  2. Under Configure storage, use an encrypted gp3 root volume of at least 20 GiB.
  3. Under Network settings, allow no inbound traffic. The gateway listens only on the instance itself, and Session Manager does not need an open port. If you also want SSH, allow port 22 from your administrative address range only. Keep outbound HTTPS open so the gateway can reach your model providers.
  4. Under Advanced details, choose an IAM instance profile that includes the AWS managed policy AmazonSSMManagedInstanceCore, and require IMDSv2.
  5. Choose Launch instance.

3. Connect

In the EC2 console, select the instance, choose Connect, then Session Manager, then Connect. Or, from your workstation:

aws ssm start-session --target <instance-id>

Switch to the ubuntu user:

sudo -iu ubuntu

4. Wait for first boot

On first boot the instance creates its certificate, master key and configuration, then starts the gateway. Wait for it to finish:

cloud-init status --wait
systemctl is-active sushi-litellm-proxy-firstboot litellm-proxy

Both services report active when the gateway is up. It can take a little longer before it answers requests.

5. Run the walkthrough

The image includes a script that checks the service and listener, the health endpoints, that a wrong key is refused, and an authenticated chat completion against the mock model:

bash /opt/sushi/docs/demo.sh

It ends with SUSHI_CUSTOMER_DEMO_PASSED.

6. Send a chat completion

On the instance, call the mock model with the master key. The key is read from the root-only file and is not printed:

sudo bash -c 'curl -sS --cacert /var/lib/litellm-proxy/tls/cert.pem \
  -H "Authorization: Bearer $(cat /var/lib/litellm-proxy/master.key)" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"smoke-mock\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}" \
  https://127.0.0.1:4000/v1/chat/completions'

The reply is a standard chat.completion object whose message content is sushi-litellm-proxy-smoke-ok.

Connect from your workstation

  1. Copy the certificate to your workstation, so your client can trust it. It is public, but readable only by root and the service:

    sudo cat /var/lib/litellm-proxy/tls/cert.pem

  2. Read the master key once, for your own secret store:

    sudo cat /var/lib/litellm-proxy/master.key

  3. On your workstation, start a port-forwarding session. Replace <instance-id> and use the instance's region:

    aws ssm start-session \
  --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["4000"],"localPortNumber":["4000"]}'

  4. Leave it running and point your OpenAI-compatible client at:

Setting Value
Base URL https://localhost:4000/v1
API key the master key
CA certificate the cert.pem you copied in step 1
Model smoke-mock, or a model name you add in Configuration

For example, with curl:

curl -sS --cacert cert.pem \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"smoke-mock","messages":[{"role":"user","content":"hello"}]}' \
  https://localhost:4000/v1/chat/completions

Trust the certificate, do not turn verification off

The certificate is self-signed for localhost and 127.0.0.1. Give your client the copied cert.pem instead of disabling TLS verification.

Keep the master key out of tickets, chat and screen recordings.

Next, read Configuration to add your model providers.

Last updated October 13, 2026 · Suggest an improvement