Getting started¶
Launch the CPU API, run the included arithmetic check, and call sushi-qwen
on the instance without putting the credential on the command line.
You need: an AWS account that can subscribe and launch EC2, and an instance profile that can use Session Manager. Your network must reach Systems Manager. No model-provider account is required.
1. Subscribe¶
Open the LocalAI CPU Model API listing, choose View purchase options, review the terms, and subscribe.
2. Launch the instance¶
- Choose t3.large with 8 GiB of memory. It is the only listed type.
- Use an encrypted gp3 root volume of at least 48 GiB.
- Require IMDSv2.
- Keep inbound ports 8080 and 50051 closed.
3. Connect¶
In the EC2 console, choose Connect, then Session Manager.
Wait for first boot to create the credential and the ubuntu client file.
The demo checks that an unauthenticated request is rejected, lists the
selected model, and asks the local model “What is 2 + 2?” with a 64-token
limit. A successful run prints Local model answered: 4 and a result
directory under ~/localai-demo/. A failure exits non-zero. Keep that
directory.
4. Call the API on the instance¶
The API base is http://127.0.0.1:8080/v1 and the model name is
sushi-qwen. Run the following as ubuntu in a shell that is not being
recorded. It sends the authorization header through a pipe into
curl --config -, so the credential is not a shell argument and is not
printed. Do not run the Python fragment alone, and do not enable shell
tracing or curl verbose output.
sudo -iu ubuntu
set +x
(
set -euo pipefail
umask 077
test "$(id -un)" = ubuntu
mkdir -p "$HOME/localai-curl"
result_dir=$(mktemp -d "$HOME/localai-curl/request.XXXXXXXX")
chmod 700 "$result_dir"
cd "$result_dir"
cat > request.json <<'JSON'
{"model":"sushi-qwen","messages":[{"role":"system","content":"Answer arithmetic questions with a single digit and no explanation."},{"role":"user","content":"What is 2 + 2?"}],"temperature":0,"max_tokens":64,"stream":false}
JSON
python3 - <<'PYCONFIG' | curl --disable --config - \
--noproxy '*' --proto '=http' --max-redirs 0 \
--connect-timeout 5 --max-time 180 --fail --silent --show-error \
--header 'Content-Type: application/json' \
--data-binary @request.json --output response.json \
http://127.0.0.1:8080/v1/chat/completions
import json, os, re, stat
from pathlib import Path
p = Path.home() / '.config/localai/client.json'
try:
info = p.lstat()
if (not stat.S_ISREG(info.st_mode) or info.st_nlink != 1
or info.st_uid != os.getuid() or info.st_mode & 0o077
or info.st_size > 4096):
raise ValueError()
client = json.loads(p.read_bytes())
if (client.get('endpoint') != 'http://127.0.0.1:8080'
or client.get('model') != 'sushi-qwen'
or re.fullmatch('[a-f0-9]{64}', client.get('api_key', '')) is None):
raise ValueError()
except Exception:
raise SystemExit('Client configuration invalid; original preserved.') from None
print('header = ' + json.dumps('Authorization: Bearer ' + client['api_key']))
PYCONFIG
python3 - <<'PYRESPONSE'
import json
from pathlib import Path
response = json.loads(Path('response.json').read_bytes())
print(response['choices'][0]['message']['content'])
PYRESPONSE
printf 'Saved request and response: %s\n' "$result_dir"
)
The block prints the message content and saves the request and response
under ~/localai-curl/. The installed demo is the check that the answer is
4. Review any other completion before you use it.
Opening http://127.0.0.1:8080 in a browser does not show a chat page.
Last updated November 11, 2026 · Suggest an improvement