Skip to content

Configuration

The API stays on the instance. Authentication is the per-instance key in the ubuntu client file.

Where things live

Item What it is
http://127.0.0.1:8080/v1 OpenAI-compatible API. Model name sushi-qwen.
Port 50051 Backend. Keep it closed to inbound traffic.
~/.config/localai/client.json Endpoint, model name, and api_key. Mode 0600.
/etc/localai Service identity.
/var/lib/localai Mutable state and caches.
/opt/sushi/docs/README.md The installed workflow.

The client file must be a regular file, one link, owned by ubuntu, mode with no group or other permissions, and at most 4,096 bytes. The endpoint value is http://127.0.0.1:8080, the model is sushi-qwen, and the key is 64 hex characters. The call in Getting started checks those conditions and leaves the original file in place when they fail.

Workstation tunnel

Install the AWS CLI and the Session Manager plugin with an identity that can start a session. Keep the command running:

aws ssm start-session --region <region> --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["8080"],"localPortNumber":["8080"]}'

The local base is then http://127.0.0.1:8080/v1. Authentication is still required. The curl example runs on the instance, where the client file stays. If you build a separate workstation client, move the credential with your own secret handling. Do not paste it into a shell command or a URL.

Do not forward port 50051.

What you cannot turn on

Galleries, automatic backend downloads, agents, MCP, and runtime settings changes are disabled. The model files are read-only. Concurrency is one inference, with two CPU threads and a 2,048-token context.

Last updated November 11, 2026 · Suggest an improvement