Skip to content

vLLM GPU model API

AI Platform & Data Hardened AMI

Serve your own compatible language model on one NVIDIA L4 with vLLM 0.32.0. The API is OpenAI-compatible and binds to loopback. You select the model after startup; no model weights, API key or chat UI are baked into the image.

Availability

Package 0.1.4 is not publicly available on AWS Marketplace. These pages preview its implemented configuration and startup behavior. Fresh-instance GPU and model validation is still pending; these instructions are not an assertion of a validated public delivery or model performance.

Delivery
Amazon Machine Image in your own AWS account

Quick start

Full guide →
  1. Connect and wait for the runtime
  2. Select a model
  3. Generate and check saved responses
  4. Connect your API client

Startup and installed components

The image supplies Ubuntu 24.04 x86_64, the GPU host tools, configuration helpers and notices. On first boot it automatically downloads the original digest-pinned vLLM 0.32.0 runtime and its PyTorch/CUDA dependencies directly from Docker Hub. The runtime image is not preinstalled in the source image. Allow about 9.21 GB of compressed downloads, plus space for unpacked layers and your model.

Outbound DNS and HTTPS, vendor availability and sufficient disk space are required. No registry login, vendor subscription, additional payment method or paid inference API is required. Anonymous shared-IP rate limits can return HTTP 429. Software Sushi remains responsible for the external runtime dependency. There is no fallback to a different image. Normal serving uses the cached runtime and selected local model offline.

Resources and scope

The documented type is g6.xlarge: one NVIDIA L4, 16 GiB host RAM and at least 256 GiB gp3 root storage. Host RAM is separate from GPU memory. CPU-only instances cannot run this package. Use encrypted storage, IMDSv2 and Session Manager; leave inbound application ports closed.

The launcher uses one GPU, float16, a 2,048-token context limit, up to four active sequences and 80% GPU memory utilization. Model capacity depends on your model and workload. This configuration does not automatically tune arbitrary model architectures or provide distributed serving. The API listens on 127.0.0.1:8000; no model server runs until you select a model.

Costs and licensing

This is paid repackaged open-source software. The configured draft offer is USD 0.03 per running instance-hour for packaging and maintenance, with a five-day software-price trial for one concurrent instance. It is not a public offer yet; confirm availability and terms before any future subscription. EC2, EBS and data transfer cost extra, including during a software-price trial. No commercial upstream subscription or technical-support agreement is bundled.

You supply model files you have the right to use. An optional explicit download provides a pinned approximately 1 GB Qwen2.5-0.5B-Instruct example. No external model-provider account or paid provider API is required for local inference.

Where to go next

Upstream project

vLLM is Apache-2.0; models, NVIDIA components and other dependencies retain their own terms. Core source and notices are under /opt/vllm/originals and /opt/vllm/licenses; GPU dependency notices are under /opt/vllm/gpu-inputs/notices. The original runtime retains its own notices. Software Sushi is independent of the upstream project. Read the vLLM 0.32.0 documentation.

Last updated November 16, 2026 · Suggest an improvement