Skip to content

Getting started

Launch the workbench, run the model-free example, and optionally add a judge.

You need: an AWS account that can subscribe and launch EC2. Session Manager needs an instance profile with Session Manager permissions and outbound access to the regional endpoints. SSH is an alternative with your own key and a tight source range. No model is required for the first run.

1. Subscribe

Open the DeepEval Evaluation Workbench listing, choose View purchase options, review the terms, and subscribe.

2. Launch the instance

  1. Choose t3.medium or a larger listed type.
  2. Use an encrypted gp3 root volume of at least 24 GiB.
  3. Require IMDSv2.
  4. Leave every application port closed. This image does not open one.

3. Connect

sudo cloud-init status --wait
sudo -iu ubuntu bash /opt/sushi/docs/demo.sh

The command writes a new report under /home/ubuntu/deepeval-results, prints its path and SHA-256, reads the saved scores, and checks that the incorrect answer failed. Each run keeps its own report.

To evaluate and read as separate steps:

sudo -iu ubuntu
sushi-deepeval evaluate
ls -lt ~/deepeval-results
sushi-deepeval read /home/ubuntu/deepeval-results/<report-path>

Pass the complete report path that evaluate printed.

4. Add a judge

The answers stay the fixed synthetic fixtures. The judge scores them. As ubuntu, in an unrecorded terminal:

sushi-deepeval configure

Enter an OpenAI-compatible base URL that includes /v1, a model id that supports JSON-object chat responses, and the key at the hidden prompt. The public guide's OpenAI example is https://api.openai.com/v1. HTTPS is required for a remote provider. A localhost server may use HTTP. This example does not call the Amazon Bedrock API directly.

The command creates ~/.config/deepeval/judge.json with mode 0600 and refuses to overwrite an existing file. Keys in the file are base_url, model, and api_key.

bash /opt/sushi/docs/demo.sh --judge

The example succeeds only when the correct answer passes and the incorrect answer fails both the JSON check and the GEval factual-correctness score. This sends billable provider requests. Telemetry and automatic dotenv loading are disabled in the supplied commands.

Last updated November 11, 2026 · Suggest an improvement