Getting started¶
Launch the workbench, run the model-free example, and optionally add a judge.
You need: an AWS account that can subscribe and launch EC2. Session Manager needs an instance profile with Session Manager permissions and outbound access to the regional endpoints. SSH is an alternative with your own key and a tight source range. No model is required for the first run.
1. Subscribe¶
Open the DeepEval Evaluation Workbench listing, choose View purchase options, review the terms, and subscribe.
2. Launch the instance¶
- Choose t3.medium or a larger listed type.
- Use an encrypted gp3 root volume of at least 24 GiB.
- Require IMDSv2.
- Leave every application port closed. This image does not open one.
3. Connect¶
The command writes a new report under /home/ubuntu/deepeval-results, prints
its path and SHA-256, reads the saved scores, and checks that the incorrect
answer failed. Each run keeps its own report.
To evaluate and read as separate steps:
sudo -iu ubuntu
sushi-deepeval evaluate
ls -lt ~/deepeval-results
sushi-deepeval read /home/ubuntu/deepeval-results/<report-path>
Pass the complete report path that evaluate printed.
4. Add a judge¶
The answers stay the fixed synthetic fixtures. The judge scores them. As
ubuntu, in an unrecorded terminal:
Enter an OpenAI-compatible base URL that includes /v1, a model id that
supports JSON-object chat responses, and the key at the hidden prompt. The
public guide's OpenAI example is https://api.openai.com/v1. HTTPS is
required for a remote provider. A localhost server may use HTTP. This
example does not call the Amazon Bedrock API directly.
The command creates ~/.config/deepeval/judge.json with mode 0600 and
refuses to overwrite an existing file. Keys in the file are base_url,
model, and api_key.
The example succeeds only when the correct answer passes and the incorrect answer fails both the JSON check and the GEval factual-correctness score. This sends billable provider requests. Telemetry and automatic dotenv loading are disabled in the supplied commands.
Last updated November 11, 2026 · Suggest an improvement