Promptfoo Evaluation Workbench

Observability Promptfoo 0.124.1 Hardened AMI

Promptfoo 0.124.1 on Ubuntu 24.04 compares prompt variants, runs assertions, and stores the results in JSON and SQLite. The included example uses a local fixture, so you can see a passing suite and a failing suite before you attach a model provider.

Get started View on AWS Marketplace

AMI release: 0.1.0 (Promptfoo 0.124.1)
Recommended: t3.medium
Storage: 24 GiB+ encrypted gp3
Access: Session Manager, no inbound ports
Released: Oct 26, 2026
Delivery: Amazon Machine Image in your own AWS account

Compare prompt variants with repeatable assertions and inspect JSON and SQLite results.

Start with an included model-free example that checks passing and deliberately failing evaluations.

Configure your own model provider after launch and retain evaluation history across restarts.

Quick start

Full guide →

What is on the image¶

This guide matches Marketplace package 0.1.0 (Promptfoo 0.124.1). Check the listing for the version currently on offer.

Promptfoo 0.124.1 at /usr/local/bin/promptfoo. Node and the application files are read-only under /opt/promptfoo.
A model-free walkthrough, /opt/sushi/docs/demo.sh. Two prompt variants over two cases produce four passing assertions. A second suite fails all four and exits 100.
Evaluation state in ~/.local/share/promptfoo/promptfoo.db. Each demo also writes a directory under ~/promptfoo-results/evaluation-*.
An optional viewer on port 15500. It does not start on boot, it has no built-in sign-in, and it has no --host flag.
A Bedrock example, /opt/sushi/bin/promptfoo-run-suite --bedrock, which makes eight short calls to Amazon Nova Lite in us-east-1.
Notices under /usr/share/doc/promptfoo.

No model weights are downloaded. No enterprise features and no provider subscription are included.

Recommended instance¶

t3.medium (4 GiB of memory) with an encrypted gp3 root volume of at least 24 GiB and IMDSv2. The CLI needs no inbound port. Leave inbound application ports closed. The optional viewer is reached by forwarding port 15500.

Billing¶

The listing states a software rate of USD 0.03 per running instance-hour. Confirm the offer before you subscribe. EC2, EBS, data transfer, and model-provider usage are billed separately. The included fixture does not call a model. The Bedrock example does.

Tutorial¶

Watch on YouTube.

Where to go next¶

Getting started: launch, run the fixture, and open the viewer.
Configuration: your provider file, the Bedrock example, and the viewer unit.
Operations: the database, backups, and support.
Troubleshooting: exit 100, the viewer, and provider keys.

Upstream project¶

Promptfoo is an open source evaluation tool. Software Sushi packages this image independently and does not claim upstream or AWS endorsement.

Package notices stay with the installed packages. Root and Node notices, plus the pinned application source, are under /usr/share/doc/promptfoo. Selected Node, libsql, and sharp-libvips source archives are under sources/. Source and patch candidates are under sources/native/. Individual dependency notices are under dependency-notices/. Full GPL and LGPL terms and native package notices are under native-notices/. Native libraries are not covered by the application MIT notice alone. Some of those files are conditional source candidates.

The Promptfoo documentation describes the upstream project. The viewer in this image has the limits in Configuration.

Last updated October 27, 2026 · Suggest an improvement