DeepEval Evaluation Workbench
DeepEval 4.2.4 on Ubuntu 24.04 compares two fixed synthetic answers about a returns policy and writes a JSON report. The local example uses DeepEval's JSON correctness metric. It does not call or download a model. An optional judge sends those same fixed answers to an OpenAI-compatible model you configure.
- AMI release
- 0.1.1-r20261105 (DeepEval 4.2.4)
- Recommended
- t3.medium
- Storage
- 24 GiB+ encrypted gp3
- Access
- Session Manager, no inbound ports
- Released
- Nov 5, 2026
- Delivery
- Amazon Machine Image in your own AWS account
Run DeepEval local JSON correctness checks against explicit good and bad synthetic answers.
Optionally use your entitled OpenAI-compatible model as a separate GEval factual-correctness judge.
Keep original JSON reports, scores and thresholds on the instance with independent saved-result readback.
Quick start
Full guide →What is on the image¶
This guide matches Marketplace version 0.1.1-r20261105 (DeepEval 4.2.4). The public customer guide describes package 0.1.1 (DeepEval 4.2.4). Check the listing for the version currently on offer.
- DeepEval 4.2.4 and the
deepevalcommand onPATH. The supplied evaluator is/opt/deepeval/evaluate.py. - A model-free walkthrough,
/opt/sushi/docs/demo.sh. The correct answer states a 30-day return with a receipt. The incorrect answer claims 90 days without a receipt and is expected to fail. - Reports under
/home/ubuntu/deepeval-results, one directory per run, readable byubuntuonly. - An optional judge file,
~/.config/deepeval/judge.json, mode0600. - No web UI, no listening API, and no application port.
Confident AI hosted dashboard, commercial platform, and subscriptions are not included. No model weights are included.
Recommended instance¶
t3.medium, x86_64, with an encrypted gp3 root volume of at least 24 GiB
and IMDSv2. The supplied examples need no GPU. A judge needs outbound HTTPS
to the provider you choose. The listed types are t3.medium, t3.large,
t3.xlarge, and t3.2xlarge.
Scope and limits¶
The reader sushi-deepeval read checks this example, including the answer
text, thresholds, and saved success flags. It is not a general report
viewer. Judge scores can vary. The public guide says to use synthetic data
until you have reviewed the provider's privacy and retention terms. There
is no application inbound port.
Unresolved operating-system findings
The 0.1.1-r20261105 release note says known OS and kernel High and Critical vulnerability findings remain unresolved, and that this delivery does not claim a security fix or a security qualification.
Billing¶
The listing states USD 0.03 per running instance-hour. Confirm the offer,
including its free trial, before you subscribe. EC2, EBS, data transfer, and
model-provider usage are billed separately. The local example does not call
a model. demo.sh --judge does.
Where to go next¶
- Getting started: launch, run the fixture, and add a judge.
- Configuration: the judge file and your own evaluations.
- Operations: reports, backups, and support.
- Troubleshooting: a failed judge request.
Upstream project¶
DeepEval core is Apache-2.0. Dependencies keep their own licenses. Software
Sushi is not affiliated with Confident AI. Notices and license files are
under /opt/deepeval. See the
DeepEval documentation. Hosted
Confident AI features need their own agreement and are not part of this image.
Last updated November 11, 2026 · Suggest an improvement