Configure the vLLM model and API¶
Select local model files you are licensed to use and keep API credentials private. This reference describes the prepublication package.
Select your own model¶
Copy Transformers-compatible configuration, tokenizer and safetensors files
into a new directory under /var/lib/vllm/models. Files must be readable by
UID 2000 and directories searchable. Do not use symlinks, Python/pickle model
files or a model that needs remote Python code. The launcher does not enable
remote model code. Check architecture support and your model's license first.
For a directory you have prepared:
Start the service and verify readiness:
Stop the service before changing a running model. The helper records selected files and their hashes. A changed model fails startup validation; review the change and reselect the intended files rather than bypassing those checks. The launcher uses float16, 2,048-token context, up to four sequences, 80% GPU memory utilization and one GPU. Choose a model that fits those fixed settings.
Optional pinned example¶
sudo python3 /opt/vllm/configure.py --example selects the locked
Qwen2.5-0.5B-Instruct download. Its destination is
/var/lib/vllm/models/qwen2.5-0.5b-instruct. The helper refuses an existing final
directory; it does not overwrite your model. Interrupted downloads retain a
separate .qwen-download-* directory and do not select incomplete files. Preserve
those files during diagnosis. After resolving the cause, rerun the example only
if its final destination does not exist, or select a reviewed complete local
model with --model-dir. No model is downloaded implicitly by server startup.
Configuration and credentials¶
| Path | Purpose |
|---|---|
/etc/vllm/model.json |
Root-managed selected model and file hashes. |
/etc/vllm/api-key |
Per-instance API key, Ubuntu-owned, mode 0600. |
/var/lib/vllm/models/ |
Local model files. |
/var/lib/vllm/cache/ |
Runtime model caches. |
/home/ubuntu/vllm-results/ |
Saved walkthrough runs. |
The key is generated on first boot and persists across service restarts. Keep
/etc/vllm root-owned with group ubuntu and mode 0750. For rotation, stop the
service, replace the key through a protected file with a new 64-character random
hex value, preserve Ubuntu ownership and mode 0600, then start the service.
Never expose the value in arguments, URLs, logs or shared screens.
Client access¶
Use the Session Manager tunnel
and API base http://127.0.0.1:8000/v1. The supplied CLI reads the local key
without putting it in arguments. Other clients need their own protected
credential storage. Keep port 8000 bound to loopback and inbound access closed.
There is no browser chat application on that port or public TLS endpoint.
Last updated November 16, 2026 · Suggest an improvement