Skip to content

LiteLLM Proxy

AI Platform & Data LiteLLM 1.104.2 Hardened AMI

LiteLLM 1.104.2 on Ubuntu 24.04 LTS, ready to run as an LLM gateway in your own AWS account. Your applications call one OpenAI-compatible endpoint, and the proxy forwards each request to the provider and model you configure, so you can switch models, add fallbacks or route apps to different providers without changing client code. It listens on the instance itself over HTTPS and needs a master key. This page covers what is on the image, the defaults it ships with and how billing works.

Get started View on AWS Marketplace
AMI release
0.1.2 (LiteLLM 1.104.2)
Recommended
t3.medium
Storage
20 GiB+ encrypted gp3
Access
Session Manager, no inbound ports
Released
Oct 12, 2026
Delivery
Amazon Machine Image in your own AWS account

One OpenAI-compatible endpoint for all your apps. Switch, combine or fail over between LLM providers without code changes.

Secure by default

HTTPS on localhost, with a certificate and master key generated for each instance on first boot.

Working in minutes

A built-in mock route answers an authenticated chat completion before you add any paid provider.

Quick start

Full guide →
  1. Subscribe
  2. Launch the instance
  3. Connect
  4. Wait for first boot
  5. Run the walkthrough
  6. Send a chat completion

What is on the image

  • LiteLLM 1.104.2, the open source proxy, installed from hash-pinned Python packages in /opt/litellm-proxy/venv.
  • A systemd service, litellm-proxy, that starts at boot and runs as the unprivileged litellm user.
  • First-boot setup that creates this instance's own TLS certificate and a private master key. Nothing secret is baked into the image.
  • A mock model, smoke-mock, that answers an authenticated chat completion with a fixed reply, so you can test the gateway before you add a paid provider.
  • A walkthrough script, /opt/sushi/docs/demo.sh, that checks the gateway end to end.

No provider credentials are included. The proprietary litellm-enterprise package is not installed. No database is configured.

Defaults

Setting Default
Endpoint https://127.0.0.1:4000/v1 (HTTPS, loopback only)
Authentication Authorization: Bearer <master key>
Master key /var/lib/litellm-proxy/master.key, root only
Certificate Self-signed for localhost and 127.0.0.1, in /var/lib/litellm-proxy/tls/
Models smoke-mock only, until you add your providers
Health /health/liveliness and /health/readiness, no key needed

Ports

Port Protocol Listens on Purpose
4000 HTTPS 127.0.0.1 OpenAI-compatible API and health checks

No inbound security group rule is needed. You reach the gateway from your workstation with Session Manager port forwarding.

t3.medium (2 vCPU, 4 GiB of memory) with an encrypted gp3 root volume of at least 20 GiB and IMDSv2. The instance needs outbound HTTPS to reach your model providers.

Billing

You pay an hourly software fee while the instance runs, as shown in the offer terms on the listing, which also shows any free trial. AWS bills EC2, EBS and data transfer separately. Your model providers bill their own usage.

Where to go next

  • Getting started: subscribe, launch, connect and send your first chat completion.
  • Configuration: add providers, store their keys and rotate the master key.
  • Operations: security model, service management, backups, updates and support.
  • Troubleshooting: common errors and how to fix them.

Upstream project

LiteLLM is developed by BerriAI and published under the MIT license. See the LiteLLM proxy documentation for providers, routing, fallbacks and the full configuration reference. Software Sushi packages the image; BerriAI does not endorse it, and its support and enterprise features are not included.

Last updated October 14, 2026 · Suggest an improvement