continuum-evaluation

Continuum Evaluation Skill

Continuum ships an evaluation framework under continuum.evaluation. It's an optional extra — install with:

pip install "shyftlabs-continuum[eval]"
# adds: deepeval >= 1.0.0, ragas >= 0.2.0

Authoritative source: src/continuum/evaluation/ in this repository. There is no dedicated user-facing doc; this skill is the primary reference.

---

Imports

from continuum.evaluation import (
    EvaluatorAgent,                   # specialised agent that scores other agents
    DeepEvalEvaluator,                # DeepEval criterion-based evaluation
    RagasEvaluator,                   # RAGAS metrics for RAG pipelines
    LangfuseDatasetClient,            # pull/push datasets from Langfuse
    EvalCase, EvalResult, EvalStatus, CriterionScore,
)

---

EvaluatorAgent

A specialised BaseAgent that takes another agent's output (plus optional reference) and produces a structured score.

from continuum.agent import BaseAgent, AgentRunner
from continuum.evaluation import EvaluatorAgent

target_agent = BaseAgent(name="target", instructions="Answer concisely.")
evaluator = EvaluatorAgent(
    name="quality-judge",
    criteria=[
        "Factual accuracy (0-1)",
        "Conciseness (0-1)",
        "Helpfulness (0-1)",
    ],
    model="gpt-4o-mini",
)

# Evaluate one trace
target_resp = await AgentRunner().run(target_agent, "What is the capital of France?")
score = await evaluator.evaluate(
    input="What is the capital of France?",
    output=target_resp.content,
    reference="Paris is the capital of France.",
)
print(score)         # {"factual_accuracy": 1.0, "conciseness": 0.9, "helpfulness": 0.8}

---

DeepEval

Use for criterion-based per-case scoring; great for unit-test-style agent regression suites.

from continuum.evaluation import DeepEvalEvaluator
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

evaluator = DeepEvalEvaluator(
    metrics=[
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.8),
    ],
)

results = await evaluator.evaluate_batch([
    {"input": "...", "output": "...", "expected_output": "...", "context": ["..."]},
    # …
])
for r in results:
    print(r.metric_name, r.score, r.success, r.reason)