llm-eval

{
  "name": "llm-eval",
  "displayName": "LLM Evaluation (DeepEval)",
  "version": "0.10.0",
  "description": "Prove an LLM/RAG/agent feature is correct with numbers a release can gate on — author eval suites in DeepEval with the metric definitions baked in: faithfulness and answer relevancy (generator), contextual precision/recall/relevancy (retriever), hallucination against ground truth, tool-call correctness for agents, plus custom G-Eval criteria, bias, and toxicity. Builds a versioned golden dataset FIRST, picks the metric that matches the actual failure mode (RAG hallucination -> faithfulness; retrieval miss -> contextual recall; wrong tool -> tool correctness), and turns every threshold into a pytest / `deepeval test run` exit code so a regression blocks the pipeline. Enforces guardrails against unpinned LLM-as-judge models, exact-string-matching non-deterministic output, contaminated goldens that leak the prompt's own examples, single-run scores with no variance, averaging unrelated metrics into one number that hides a failing dimension, and getting metric direction wrong (since deepeval 4.2.0 hallucination/bias/toxicity score the clean share and pass at score >= threshold; older versions inverted that, so the version is pinned and thresholds re-baselined across the change). Adds an adversarial/safety slice (prompt injection, jailbreak, PII) and judge calibration against human labels.",
  "author": {
    "name": "pnakhat"
  },
  "homepage": "https://github.com/pnakhat/qa-ai-repo/tree/main/llm-eval",
  "repository": "https://github.com/pnakhat/qa-ai-repo",
  "license": "MIT",
  "keywords": [
    "qa",
    "testing",
    "llm-eval"
  ]
}