Configuration¶
The API stays on the instance. Authentication is the per-instance key in the
ubuntu client file.
Where things live¶
| Item | What it is |
|---|---|
http://127.0.0.1:8080/v1 |
OpenAI-compatible API. Model name sushi-qwen. |
| Port 50051 | Backend. Keep it closed to inbound traffic. |
~/.config/localai/client.json |
Endpoint, model name, and api_key. Mode 0600. |
/etc/localai |
Service identity. |
/var/lib/localai |
Mutable state and caches. |
/opt/sushi/docs/README.md |
The installed workflow. |
The client file must be a regular file, one link, owned by ubuntu, mode
with no group or other permissions, and at most 4,096 bytes. The endpoint
value is http://127.0.0.1:8080, the model is sushi-qwen, and the key is
64 hex characters. The call in
Getting started checks
those conditions and leaves the original file in place when they fail.
Workstation tunnel¶
Install the AWS CLI and the Session Manager plugin with an identity that can start a session. Keep the command running:
aws ssm start-session --region <region> --target <instance-id> \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["8080"],"localPortNumber":["8080"]}'
The local base is then http://127.0.0.1:8080/v1. Authentication is still
required. The curl example runs on the instance, where the client file
stays. If you build a separate workstation client, move the credential with
your own secret handling. Do not paste it into a shell command or a URL.
Do not forward port 50051.
What you cannot turn on¶
Galleries, automatic backend downloads, agents, MCP, and runtime settings changes are disabled. The model files are read-only. Concurrency is one inference, with two CPU threads and a 2,048-token context.
Last updated November 11, 2026 · Suggest an improvement