JevEval

JevEval is an evaluation metric that Confident AI added to DeepEval, its open-source evaluation framework, for scoring the answers an AI app gives. You write a set of explicit questions, such as whether the response makes unsupported claims; the Jev model returns a probability for each one in a single call, and DeepEval turns those into a score with a fixed formula. Questions come in three kinds: yes/no, graded scales and multiple-choice behaviors, and each can carry a weight. Compared with the usual LLM-as-a-judge setup, the official blog says JevEval separates the steps: you define the criteria, Jev makes the decisions and the formula computes the score, so the same probabilities always give the same score, and the reason text is assembled from the outcomes. For multi-turn conversations there's ConversationalJevEval.

Who it's for:development teams building AI apps who need automated tests for model output. Readers who don't write code won't have a use for it.

Before you start: install with pip install -U deepeval typesafe-sdk, and you'll need a TypeSafe API key (TYPESAFE_API_KEY). Each evaluation sends the test fields you choose, such as the input, the model's answer and the tools it called, along with your questions to Jev; the blog doesn't say how that data is stored. Pricing isn't in the blog; check the official site. DeepEval sends anonymous usage telemetry by default, which you can turn off with DEEPEVAL_TELEMETRY_OPT_OUT=1. DeepEval's code is Apache-2.0; the Jev model service isn't covered by that license.

Based on the official blog and the project's public documentation. Not tested by AI Inspiration World.