LocalAI CPU Model API
LocalAI 4.12.0 on Ubuntu 24.04 serves one baked model, Qwen2.5-0.5B-Instruct
in Q4_K_M, under the name sushi-qwen. The API is OpenAI-compatible and
authenticated. It listens on the instance. There is no web UI and no
external inference account. The model is in the image, so launch does not
download weights.
- AMI release
- 0.1.0 (LocalAI 4.12.0)
- Recommended
- t3.large
- Storage
- 48 GiB+ encrypted gp3
- Access
- Session Manager or SSH
- Released
- Oct 28, 2026
- Delivery
- Amazon Machine Image in your own AWS account
Call a local Qwen2.5-0.5B-Instruct Q4_K_M CPU model as sushi-qwen through authenticated text-chat endpoints, without an external inference account.
Use the included ubuntu demo to check model identity and an arithmetic completion, with original request and response files retained in a dedicated results directory.
Launch t3.large with 8 GiB RAM and 48 GiB encrypted storage; connect through Session Manager while keeping application and backend ports closed to inbound traffic.
Quick start
Full guide →What is on the image¶
This guide matches Marketplace package 0.1.0 (LocalAI 4.12.0). Check the listing for the version currently on offer.
- LocalAI 4.12.0 and the model
sushi-qwen. - HTTP API on
127.0.0.1:8080. The base path is/v1. - A backend port 50051 that must stay closed to inbound traffic.
- A per-instance credential in
~/.config/localai/client.json, mode0600, owned byubuntu. - A walkthrough,
/opt/sushi/docs/demo.sh, that rejects an unauthenticated call, checks the model name, and asks the local model what 2 + 2 is, with a 64-token limit. - Mutable state under
/var/lib/localai. Identity files are under/etc/localai.
Galleries, automatic backend downloads, agents, MCP, and runtime settings changes are disabled. The configuration allows one concurrent inference, two CPU threads, and a 2,048-token context.
Recommended instance¶
t3.large (8 GiB of memory). This is the only listed instance type. Use an
encrypted gp3 root volume of at least 48 GiB and IMDSv2. Keep inbound ports
8080 and 50051 closed. The recommended size is headroom for the operating
system, context, caches, and state. It is not a measured minimum and it is
not a throughput guarantee.
Scope and limits¶
Treat generated text as model output and review it. The included task checks one arithmetic completion. It does not measure quality on your own prompts. Model files are read-only.
Billing¶
The listing states USD 0.03 per running instance-hour. Confirm the offer, including its free trial, before you subscribe. EC2, EBS, and data transfer are billed separately. The local model does not call a paid inference API.
Where to go next¶
- Getting started: launch, run the demo, and call the API.
- Configuration: the client file and a workstation tunnel.
- Operations: backups and support.
- Troubleshooting: startup and a rejected call.
Upstream project¶
LocalAI is MIT. The selected Qwen model is Apache-2.0. Read the notices and
model card in /usr/share/doc/localai/. Software Sushi does not claim
upstream or AWS endorsement. See the
LocalAI documentation and the
public getting-started page.
Last updated November 11, 2026 · Suggest an improvement