Skip to content

Get started with vLLM

Wait for the original runtime download, select a compatible model and access its private API. The optional example downloads model files only when you ask.

Prepublication guide

Package 0.1.4 is not publicly available. Fresh-instance GPU and model validation remains pending. This guide describes the implemented setup for authorized users with access; it is not a public launch offer.

1. Connect and wait for the runtime

Use the documented g6.xlarge configuration with one L4, 16 GiB host RAM and a 256 GiB encrypted gp3 root volume. Require IMDSv2 and attach an instance role with Session Manager access. Keep inbound application ports closed. Allow the outbound DNS and HTTPS needed by Systems Manager, Docker Hub and its download endpoints. Open a Session Manager shell:

sudo cloud-init status --wait
sudo systemctl status sushi-vllm-runtime-acquire.service --no-pager
sudo journalctl -u sushi-vllm-runtime-acquire.service --no-pager

Wait for the acquisition service to finish successfully as active (exited). The automatic acquisition downloads about 9.21 GB compressed and is bounded to 30 minutes. Anonymous Docker Hub rate limits or vendor unavailability can prevent completion. Resolve the cause before rebooting for a new automatic attempt; there is no automatic fallback to another image.

2. Select a model

For the optional public Qwen2.5-0.5B-Instruct example:

sudo python3 /opt/vllm/configure.py --example

This downloads approximately 1 GB from Hugging Face, checks each file's locked size and SHA-256, and selects it. It needs outbound HTTPS to Hugging Face and its download endpoints, but no provider account or paid API. The example has its own Apache-2.0 license. The exact revision is in /opt/vllm/model.lock.json. Alternatively, select your own local model.

Start the server, then wait for model readiness:

sudo systemctl start vllm.service
sudo -iu ubuntu /usr/local/bin/sushi-vllm wait

First model loading and kernel compilation can take several minutes. The wait command allows up to 15 minutes. No chat UI is provided.

3. Generate and check saved responses

sudo -iu ubuntu bash /opt/sushi/docs/demo.sh

The synthetic walkthrough checks rejection of unauthenticated requests, a JSON generation, a streaming response and two concurrent requests. It recomputes expected arithmetic totals, retains original responses and their hashes in a new directory under /home/ubuntu/vllm-results/, and independently rereads them. Success prints SUSHI_CUSTOMER_DEMO_PASSED. A selected model that cannot follow the task causes failure; these examples are not quality or throughput benchmarks.

4. Connect your API client

On your workstation, use an authorized AWS identity, AWS CLI and Session Manager plugin. Leave this tunnel open:

read -r -p 'EC2 instance ID: ' vllm_instance_id
read -r -p 'AWS region: ' vllm_region
aws ssm start-session --region "$vllm_region" --target "$vllm_instance_id" \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["8000"],"localPortNumber":["8000"]}'

Set your client's API base to http://127.0.0.1:8000/v1. Load the instance key from /etc/vllm/api-key through a protected file or your client's credential store; do not put it in arguments, URLs, history or screenshots. The supplied instance CLI already reads that private file. /v1/models lists the selected served model. See configuration for supported model files.

Last updated November 16, 2026 · Suggest an improvement