Get started with vLLM¶
Wait for the original runtime download, select a compatible model and access its private API. The optional example downloads model files only when you ask.
Prepublication guide
Package 0.1.4 is not publicly available. Fresh-instance GPU and model validation remains pending. This guide describes the implemented setup for authorized users with access; it is not a public launch offer.
1. Connect and wait for the runtime¶
Use the documented g6.xlarge configuration with one L4, 16 GiB host RAM and a
256 GiB encrypted gp3 root volume. Require IMDSv2 and attach an instance role
with Session Manager access. Keep inbound application ports closed. Allow the
outbound DNS and HTTPS needed by Systems Manager, Docker Hub and its download
endpoints. Open a Session Manager shell:
sudo cloud-init status --wait
sudo systemctl status sushi-vllm-runtime-acquire.service --no-pager
sudo journalctl -u sushi-vllm-runtime-acquire.service --no-pager
Wait for the acquisition service to finish successfully as active (exited).
The automatic acquisition downloads about 9.21 GB compressed and is bounded to
30 minutes. Anonymous Docker Hub rate limits or vendor unavailability can prevent
completion. Resolve the cause before rebooting for a new automatic attempt;
there is no automatic fallback to another image.
2. Select a model¶
For the optional public Qwen2.5-0.5B-Instruct example:
This downloads approximately 1 GB from Hugging Face, checks each file's locked
size and SHA-256, and selects it. It needs outbound HTTPS to Hugging Face and
its download endpoints, but no provider account or paid API. The example has
its own Apache-2.0 license. The exact revision is in /opt/vllm/model.lock.json.
Alternatively, select your own local model.
Start the server, then wait for model readiness:
First model loading and kernel compilation can take several minutes. The wait command allows up to 15 minutes. No chat UI is provided.
3. Generate and check saved responses¶
The synthetic walkthrough checks rejection of unauthenticated requests, a JSON
generation, a streaming response and two concurrent requests. It recomputes
expected arithmetic totals, retains original responses and their hashes in a
new directory under /home/ubuntu/vllm-results/, and independently rereads them.
Success prints SUSHI_CUSTOMER_DEMO_PASSED. A selected model that cannot follow
the task causes failure; these examples are not quality or throughput benchmarks.
4. Connect your API client¶
On your workstation, use an authorized AWS identity, AWS CLI and Session Manager plugin. Leave this tunnel open:
read -r -p 'EC2 instance ID: ' vllm_instance_id
read -r -p 'AWS region: ' vllm_region
aws ssm start-session --region "$vllm_region" --target "$vllm_instance_id" \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["8000"],"localPortNumber":["8000"]}'
Set your client's API base to http://127.0.0.1:8000/v1. Load the instance key
from /etc/vllm/api-key through a protected file or your client's credential
store; do not put it in arguments, URLs, history or screenshots. The supplied
instance CLI already reads that private file. /v1/models lists the selected
served model. See configuration for supported model files.
Last updated November 16, 2026 · Suggest an improvement