llm-eval

LLM Eval with deepeval

When to Use deepeval vs Spot-Checking

Rule of thumb: if you'd manually check the same thing twice, automate it.

Installation

pip install -U deepeval langfuse
deepeval login   # optional: connects to Confident AI dashboard

Core Metrics — Use These First

from deepeval.metrics import (
    GEval,
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    HallucinationMetric,
)
from deepeval.test_case import LLMTestCase, SingleTurnParams

# 1. G-Eval — custom rubric (SME-locked criteria go here)
sme_rubric = GEval(
    name="CurriculumCorrectness",
    criteria=(
        "The answer must be factually accurate per NCERT Class 10 syllabus. "
        "Penalise any out-of-syllabus content or incorrect formulae."
    ),
    evaluation_params=[
        SingleTurnParams.INPUT,
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.EXPECTED_OUTPUT,
    ],
    threshold=0.75,
)

# 2. Answer Relevancy — does the response address the question?
relevancy = AnswerRelevancyMetric(threshold=0.7)

# 3. Faithfulness — response grounded in retrieved context (RAG)
faithfulness = FaithfulnessMetric(threshold=0.8)

# 4. Hallucination — detects contradictions with context
hallucination = HallucinationMetric(threshold=0.2)  # lower = stricter

Dataset Format

LLMTestCase(
    input="What is the formula for kinetic energy?",          # user query
    actual_output=your_llm_response,                          # live LLM call
    expected_output="KE = 0.5 * m * v^2",                    # golden answer
    retrieval_context=["Kinetic energy is the energy ..."],   # RAG chunks; omit if non-RAG
)