Skip to content

Troubleshoot vLLM

Separate first-boot runtime acquisition from model loading and inference. This reference describes the prepublication package, whose fresh-instance GPU and model validation is still pending.

The runtime download has not finished

Inspect the acquisition service:

sudo systemctl status sushi-vllm-runtime-acquire.service --no-pager
sudo journalctl -u sushi-vllm-runtime-acquire.service --no-pager

Allow for about 9.21 GB compressed plus unpacked disk space. Verify outbound DNS, HTTPS and vendor download availability. Anonymous Docker Hub shared-IP rate limits can produce HTTP 429; there is no fixed guaranteed allowance or fallback image. Acquisition is bounded to 30 minutes. Resolve connectivity, availability or storage problems before rebooting for the next automatic attempt. Do not manually substitute a runtime image.

The API is not ready

Select a model first, start vllm.service and run the supplied wait command. First loading and compilation can take several minutes; the command allows up to 15 minutes. Check GPU visibility and the service journal:

nvidia-smi
sudo journalctl -u vllm.service --since '15 minutes ago' --no-pager

Use the documented GPU instance. CPU-only instances cannot run the server. For memory errors, choose a smaller compatible model within the fixed context, sequence and GPU-memory settings. Other architectures are not tuned automatically.

Model selection or startup validation fails

Use a physical local directory with readable config, tokenizer and safetensors files. Symlinks, Python/pickle files and remote model code are not supported. If files changed after selection, review them and their license and select the intended model again. Do not remove hash checks. The example helper refuses an existing final directory and preserves incomplete downloads separately; see the example recovery notes.

Requests return 401 or the client cannot connect

Use the instance key through a protected file or client credential store. The supplied CLI reads it automatically. Confirm that the Session Manager tunnel is open and that your API base is http://127.0.0.1:8000/v1. Port 8000 is an API, not a browser chat page. Keep inbound access closed rather than exposing it to work around a tunnel problem.

The saved walkthrough fails

The chosen model must follow the synthetic JSON and arithmetic instructions. A model that does not follow them fails the task. Keep the original responses and manifest; reread a completed result without new inference. These examples do not establish model accuracy, throughput or suitability for your workload.

Last updated November 16, 2026 · Suggest an improvement