Skip to content

Configure the vLLM model and API

Select local model files you are licensed to use and keep API credentials private. This reference describes the prepublication package.

Select your own model

Copy Transformers-compatible configuration, tokenizer and safetensors files into a new directory under /var/lib/vllm/models. Files must be readable by UID 2000 and directories searchable. Do not use symlinks, Python/pickle model files or a model that needs remote Python code. The launcher does not enable remote model code. Check architecture support and your model's license first.

For a directory you have prepared:

sudo python3 /opt/vllm/configure.py \
  --model-dir /var/lib/vllm/models/my-model --name sushi-model

Start the service and verify readiness:

sudo systemctl start vllm.service
sudo -iu ubuntu /usr/local/bin/sushi-vllm wait

Stop the service before changing a running model. The helper records selected files and their hashes. A changed model fails startup validation; review the change and reselect the intended files rather than bypassing those checks. The launcher uses float16, 2,048-token context, up to four sequences, 80% GPU memory utilization and one GPU. Choose a model that fits those fixed settings.

Optional pinned example

sudo python3 /opt/vllm/configure.py --example selects the locked Qwen2.5-0.5B-Instruct download. Its destination is /var/lib/vllm/models/qwen2.5-0.5b-instruct. The helper refuses an existing final directory; it does not overwrite your model. Interrupted downloads retain a separate .qwen-download-* directory and do not select incomplete files. Preserve those files during diagnosis. After resolving the cause, rerun the example only if its final destination does not exist, or select a reviewed complete local model with --model-dir. No model is downloaded implicitly by server startup.

Configuration and credentials

Path Purpose
/etc/vllm/model.json Root-managed selected model and file hashes.
/etc/vllm/api-key Per-instance API key, Ubuntu-owned, mode 0600.
/var/lib/vllm/models/ Local model files.
/var/lib/vllm/cache/ Runtime model caches.
/home/ubuntu/vllm-results/ Saved walkthrough runs.

The key is generated on first boot and persists across service restarts. Keep /etc/vllm root-owned with group ubuntu and mode 0750. For rotation, stop the service, replace the key through a protected file with a new 64-character random hex value, preserve Ubuntu ownership and mode 0600, then start the service. Never expose the value in arguments, URLs, logs or shared screens.

Client access

Use the Session Manager tunnel and API base http://127.0.0.1:8000/v1. The supplied CLI reads the local key without putting it in arguments. Other clients need their own protected credential storage. Keep port 8000 bound to loopback and inbound access closed. There is no browser chat application on that port or public TLS endpoint.

Last updated November 16, 2026 · Suggest an improvement