Skip to content

Operate vLLM

Manage model serving, protect stored keys and preserve the original saved responses. This reference describes the prepublication package; fresh-instance GPU and model validation is pending.

Services and GPU diagnostics

Check the server and recent logs:

sudo systemctl status vllm.service --no-pager
sudo journalctl -u vllm.service --since '15 minutes ago' --no-pager

Check GPU and driver visibility:

nvidia-smi

Stop serving before changing a model, credentials or taking a consistent backup:

sudo systemctl stop vllm.service

After configuration or backup, start and wait:

sudo systemctl start vllm.service
sudo -iu ubuntu /usr/local/bin/sushi-vllm wait

The automatic first-boot acquisition is a separate service, sushi-vllm-runtime-acquire.service. Inspect its journal for runtime download errors before starting the model server. Do not substitute a mutable image or manually pull a different runtime to bypass acquisition checks.

Reread saved responses

Switch to ubuntu:

sudo -iu ubuntu

Reread an existing directory without new inference:

read -r -p 'Full saved result directory: ' vllm_saved_run
/usr/local/bin/sushi-vllm read --output "$vllm_saved_run"

Keep the manifest, JSON generation response, streaming response and concurrent responses together. The checker verifies hashes and recomputes expected totals. A failed example remains useful diagnostic output; do not edit it into a pass.

Persistence and backups

Back up /etc/vllm/, /var/lib/vllm/models/, /var/lib/vllm/cache/ and /home/ubuntu/vllm-results/ to encrypted storage with restricted readers. The cached runtime resides in /var/lib/docker; retaining the complete root EBS volume also preserves it. Allow enough storage for unpacked runtime layers, models and caches. Termination can delete the root volume.

Normal serving uses the cached runtime and selected local model without fetching new weights. Keep endpoint access private and review logs and saved responses for sensitive content before sharing. Your own prompts and models can contain confidential data even though the supplied example uses synthetic requests.

Updates and support

Use a reviewed replacement image for runtime or GPU dependency updates. Arbitrary pip upgrades or image pulls can invalidate the managed configuration. Model changes require license review and reselection through the supplied helper.

Contact Software Sushi with the package version, symptom and redacted service output. Never send the API key, private model files or prompts. Technical-support coverage requires a separate agreement; commercial upstream support is not bundled.

Last updated November 16, 2026 · Suggest an improvement