Skip to content
Promptfoo Evaluation Workbench Observability · Promptfoo 0.124.1 Launch on AWS

Getting started

Launch the Promptfoo Evaluation Workbench, run the model-free example, and open the optional viewer through a tunnel.

You need: an AWS account that can subscribe on AWS Marketplace, launch EC2, and start Session Manager. The viewer forward needs the AWS CLI and the Session Manager plugin on your workstation. No key pair and no inbound rule are required for that forward.

Watch on YouTube.

1. Subscribe

Open the Promptfoo Evaluation Workbench listing, choose View purchase options, review the terms, and subscribe. When the subscription is active, choose Launch your software.

2. Launch the instance

On the EC2 launch page:

  1. Choose t3.medium (4 GiB of memory) or larger.
  2. Under Configure storage, use an encrypted gp3 root volume of at least 24 GiB.
  3. Require IMDSv2.
  4. Attach an IAM instance profile that includes AmazonSSMManagedInstanceCore. Add bedrock:InvokeModel only if you run the Nova Lite example later.
  5. Leave inbound application ports closed.
  6. Choose Launch instance.

3. Connect

Session Manager starts as ssm-user. In the EC2 console, choose Connect, then Session Manager, then Connect. Or:

aws ssm start-session --target <instance-id>

Wait for cloud-init, then run the walkthrough as ubuntu:

sudo cloud-init status --wait
sudo -iu ubuntu bash /opt/sushi/docs/demo.sh

4. Run the walkthrough

The demo sends evaluation requests to a bounded localhost fixture. It uses no model account and downloads no weights.

Two prompt variants over two cases produce four passing assertions. A second suite fails all four on purpose. Promptfoo's exit code for that suite is 100. The command checks both result files and the SQLite database. It writes a directory under ~/promptfoo-results/evaluation-*.

On every later run, both earlier evaluation IDs and the unchanged JSON must still be present before a new evaluation starts. A failed assertion is an evaluation result. An installer or provider error fails the command separately.

First boot keeps evaluation state that is already on disk.

5. Evaluate your own model

Point Promptfoo at your config file:

promptfoo eval --config /path/to/your/promptfooconfig.yaml

Put provider credentials in a file readable only by ubuntu. Never pass keys as command arguments. Your provider may charge for the requests.

For an instance role that may call Amazon Nova Lite in us-east-1:

/opt/sushi/bin/promptfoo-run-suite --bedrock

That command makes eight short model requests and checks the responses. The role must allow bedrock:InvokeModel for amazon.nova-lite-v1:0, and the model must be available in the account. Model availability is your prerequisite.

6. Open the viewer

The viewer does not start with the instance. It has no built-in sign-in, it listens on network interfaces, and it has no --host flag. Start the bounded unit below from an administrative Session Manager shell. It runs as ubuntu, allows network traffic only inside the instance, and stops after one hour.

sudo systemd-run --unit=promptfoo-viewer --uid=ubuntu \
  --setenv=HOME=/home/ubuntu \
  --property=RuntimeMaxSec=3600 \
  --property=IPAddressDeny=any \
  --property=IPAddressAllow=localhost \
  /usr/local/bin/promptfoo view --no --port 15500

Stop it sooner with sudo systemctl stop promptfoo-viewer. Other local users on the instance can still open its results. Use an instance limited to people who may see those evaluations.

From your workstation, forward port 15500. Replace <region> with the instance's Region:

aws ssm start-session --region <region> --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["15500"],"localPortNumber":["15500"]}'

Open http://127.0.0.1:15500. The image includes no viewer password. The tunnel is the remote access control. Do not publish the viewer on the internet.

Next, see Configuration for where the database lives.

Last updated October 27, 2026 · Suggest an improvement