Skip to content

Configuration

There is no service to reconfigure and no port to forward. Access is the ubuntu account on the instance.

Where things live

Item What it is
deepeval Upstream CLI, on PATH.
sushi-deepeval evaluate, read, and configure for the supplied example.
/opt/deepeval/evaluate.py The two example criteria.
/opt/deepeval/venv/bin/python Import deepeval from your own workspace.
~/deepeval-results/ One report directory per run.
~/.config/deepeval/judge.json Optional judge. Mode 0600. Keys base_url, model, api_key.
/var/lib/deepeval Initialized state. Do not delete it to repair a run.

Your own evaluations

Run deepeval --help for the upstream commands. Use the virtualenv Python to create LLMTestCase values and select metrics. Keep the installed runtime intact and use a separate environment when you add dependencies. Features beyond the supplied example can need extra packages, a provider, or a Confident AI account. Those are not included.

Judge file

sushi-deepeval configure will not replace an existing file. Edit judge.json locally to change provider, model, or key. Do not paste the key into a shell command, a log, a screenshot, or a support ticket. chmod 600 ~/.config/deepeval/judge.json restores the mode if it was widened.

The application under test and the judge are separate. The supplied answers are fixtures. A judge request sends the question, the actual answer, and the reference answer to your provider.

Last updated November 11, 2026 · Suggest an improvement