Skip to content

LocalAI CPU Model API

AI Platform & Data LocalAI 4.12.0 Hardened AMI

LocalAI 4.12.0 on Ubuntu 24.04 serves one baked model, Qwen2.5-0.5B-Instruct in Q4_K_M, under the name sushi-qwen. The API is OpenAI-compatible and authenticated. It listens on the instance. There is no web UI and no external inference account. The model is in the image, so launch does not download weights.

AMI release
0.1.0 (LocalAI 4.12.0)
Recommended
t3.large
Storage
48 GiB+ encrypted gp3
Access
Session Manager or SSH
Released
Oct 28, 2026
Delivery
Amazon Machine Image in your own AWS account

Call a local Qwen2.5-0.5B-Instruct Q4_K_M CPU model as sushi-qwen through authenticated text-chat endpoints, without an external inference account.

Use the included ubuntu demo to check model identity and an arithmetic completion, with original request and response files retained in a dedicated results directory.

Launch t3.large with 8 GiB RAM and 48 GiB encrypted storage; connect through Session Manager while keeping application and backend ports closed to inbound traffic.

Quick start

Full guide →
  1. Subscribe
  2. Launch the instance
  3. Connect
  4. Call the API on the instance

What is on the image

This guide matches Marketplace package 0.1.0 (LocalAI 4.12.0). Check the listing for the version currently on offer.

  • LocalAI 4.12.0 and the model sushi-qwen.
  • HTTP API on 127.0.0.1:8080. The base path is /v1.
  • A backend port 50051 that must stay closed to inbound traffic.
  • A per-instance credential in ~/.config/localai/client.json, mode 0600, owned by ubuntu.
  • A walkthrough, /opt/sushi/docs/demo.sh, that rejects an unauthenticated call, checks the model name, and asks the local model what 2 + 2 is, with a 64-token limit.
  • Mutable state under /var/lib/localai. Identity files are under /etc/localai.

Galleries, automatic backend downloads, agents, MCP, and runtime settings changes are disabled. The configuration allows one concurrent inference, two CPU threads, and a 2,048-token context.

t3.large (8 GiB of memory). This is the only listed instance type. Use an encrypted gp3 root volume of at least 48 GiB and IMDSv2. Keep inbound ports 8080 and 50051 closed. The recommended size is headroom for the operating system, context, caches, and state. It is not a measured minimum and it is not a throughput guarantee.

Scope and limits

Treat generated text as model output and review it. The included task checks one arithmetic completion. It does not measure quality on your own prompts. Model files are read-only.

Billing

The listing states USD 0.03 per running instance-hour. Confirm the offer, including its free trial, before you subscribe. EC2, EBS, and data transfer are billed separately. The local model does not call a paid inference API.

Where to go next

Upstream project

LocalAI is MIT. The selected Qwen model is Apache-2.0. Read the notices and model card in /usr/share/doc/localai/. Software Sushi does not claim upstream or AWS endorsement. See the LocalAI documentation and the public getting-started page.

Last updated November 11, 2026 · Suggest an improvement