Skip to content

Getting started

Launch the CPU API, run the included arithmetic check, and call sushi-qwen on the instance without putting the credential on the command line.

You need: an AWS account that can subscribe and launch EC2, and an instance profile that can use Session Manager. Your network must reach Systems Manager. No model-provider account is required.

1. Subscribe

Open the LocalAI CPU Model API listing, choose View purchase options, review the terms, and subscribe.

2. Launch the instance

  1. Choose t3.large with 8 GiB of memory. It is the only listed type.
  2. Use an encrypted gp3 root volume of at least 48 GiB.
  3. Require IMDSv2.
  4. Keep inbound ports 8080 and 50051 closed.

3. Connect

In the EC2 console, choose Connect, then Session Manager.

sudo cloud-init status --wait
sudo -iu ubuntu bash /opt/sushi/docs/demo.sh

Wait for first boot to create the credential and the ubuntu client file. The demo checks that an unauthenticated request is rejected, lists the selected model, and asks the local model “What is 2 + 2?” with a 64-token limit. A successful run prints Local model answered: 4 and a result directory under ~/localai-demo/. A failure exits non-zero. Keep that directory.

4. Call the API on the instance

The API base is http://127.0.0.1:8080/v1 and the model name is sushi-qwen. Run the following as ubuntu in a shell that is not being recorded. It sends the authorization header through a pipe into curl --config -, so the credential is not a shell argument and is not printed. Do not run the Python fragment alone, and do not enable shell tracing or curl verbose output.

sudo -iu ubuntu
set +x
(
set -euo pipefail
umask 077
test "$(id -un)" = ubuntu
mkdir -p "$HOME/localai-curl"
result_dir=$(mktemp -d "$HOME/localai-curl/request.XXXXXXXX")
chmod 700 "$result_dir"
cd "$result_dir"
cat > request.json <<'JSON'
{"model":"sushi-qwen","messages":[{"role":"system","content":"Answer arithmetic questions with a single digit and no explanation."},{"role":"user","content":"What is 2 + 2?"}],"temperature":0,"max_tokens":64,"stream":false}
JSON
python3 - <<'PYCONFIG' | curl --disable --config - \
  --noproxy '*' --proto '=http' --max-redirs 0 \
  --connect-timeout 5 --max-time 180 --fail --silent --show-error \
  --header 'Content-Type: application/json' \
  --data-binary @request.json --output response.json \
  http://127.0.0.1:8080/v1/chat/completions
import json, os, re, stat
from pathlib import Path
p = Path.home() / '.config/localai/client.json'
try:
    info = p.lstat()
    if (not stat.S_ISREG(info.st_mode) or info.st_nlink != 1
            or info.st_uid != os.getuid() or info.st_mode & 0o077
            or info.st_size > 4096):
        raise ValueError()
    client = json.loads(p.read_bytes())
    if (client.get('endpoint') != 'http://127.0.0.1:8080'
            or client.get('model') != 'sushi-qwen'
            or re.fullmatch('[a-f0-9]{64}', client.get('api_key', '')) is None):
        raise ValueError()
except Exception:
    raise SystemExit('Client configuration invalid; original preserved.') from None
print('header = ' + json.dumps('Authorization: Bearer ' + client['api_key']))
PYCONFIG
python3 - <<'PYRESPONSE'
import json
from pathlib import Path
response = json.loads(Path('response.json').read_bytes())
print(response['choices'][0]['message']['content'])
PYRESPONSE
printf 'Saved request and response: %s\n' "$result_dir"
)

The block prints the message content and saves the request and response under ~/localai-curl/. The installed demo is the check that the answer is 4. Review any other completion before you use it.

Opening http://127.0.0.1:8080 in a browser does not show a chat page.

Last updated November 11, 2026 · Suggest an improvement