Skip to content

DeepEval Evaluation Workbench

AI Platform & Data DeepEval 4.2.4 Hardened AMI

DeepEval 4.2.4 on Ubuntu 24.04 compares two fixed synthetic answers about a returns policy and writes a JSON report. The local example uses DeepEval's JSON correctness metric. It does not call or download a model. An optional judge sends those same fixed answers to an OpenAI-compatible model you configure.

AMI release
0.1.1-r20261105 (DeepEval 4.2.4)
Recommended
t3.medium
Storage
24 GiB+ encrypted gp3
Access
Session Manager, no inbound ports
Released
Nov 5, 2026
Delivery
Amazon Machine Image in your own AWS account

Run DeepEval local JSON correctness checks against explicit good and bad synthetic answers.

Optionally use your entitled OpenAI-compatible model as a separate GEval factual-correctness judge.

Keep original JSON reports, scores and thresholds on the instance with independent saved-result readback.

Quick start

Full guide →
  1. Subscribe
  2. Launch the instance
  3. Connect
  4. Add a judge

What is on the image

This guide matches Marketplace version 0.1.1-r20261105 (DeepEval 4.2.4). The public customer guide describes package 0.1.1 (DeepEval 4.2.4). Check the listing for the version currently on offer.

  • DeepEval 4.2.4 and the deepeval command on PATH. The supplied evaluator is /opt/deepeval/evaluate.py.
  • A model-free walkthrough, /opt/sushi/docs/demo.sh. The correct answer states a 30-day return with a receipt. The incorrect answer claims 90 days without a receipt and is expected to fail.
  • Reports under /home/ubuntu/deepeval-results, one directory per run, readable by ubuntu only.
  • An optional judge file, ~/.config/deepeval/judge.json, mode 0600.
  • No web UI, no listening API, and no application port.

Confident AI hosted dashboard, commercial platform, and subscriptions are not included. No model weights are included.

t3.medium, x86_64, with an encrypted gp3 root volume of at least 24 GiB and IMDSv2. The supplied examples need no GPU. A judge needs outbound HTTPS to the provider you choose. The listed types are t3.medium, t3.large, t3.xlarge, and t3.2xlarge.

Scope and limits

The reader sushi-deepeval read checks this example, including the answer text, thresholds, and saved success flags. It is not a general report viewer. Judge scores can vary. The public guide says to use synthetic data until you have reviewed the provider's privacy and retention terms. There is no application inbound port.

Unresolved operating-system findings

The 0.1.1-r20261105 release note says known OS and kernel High and Critical vulnerability findings remain unresolved, and that this delivery does not claim a security fix or a security qualification.

Billing

The listing states USD 0.03 per running instance-hour. Confirm the offer, including its free trial, before you subscribe. EC2, EBS, data transfer, and model-provider usage are billed separately. The local example does not call a model. demo.sh --judge does.

Where to go next

Upstream project

DeepEval core is Apache-2.0. Dependencies keep their own licenses. Software Sushi is not affiliated with Confident AI. Notices and license files are under /opt/deepeval. See the DeepEval documentation. Hosted Confident AI features need their own agreement and are not part of this image.

Last updated November 11, 2026 · Suggest an improvement