trulens-diagnosis

TruLens Diagnosis and Improvement

After running evaluations, use this skill to diagnose why scores are low and what to change — closing the loop from eval results to concrete fixes.

Prerequisites

Before diagnosing, ensure you have:

1. Run evaluations with at least one feedback function (see running-evaluations skill)
2. Access to the session to retrieve records and feedback results

Instructions

Step 1: Triage — identify failing metrics and patterns

Pull records with feedback scores and filter to failing cases:

import pandas as pd
from trulens.core import TruSession

session = TruSession()

# Get all records with feedback columns
records_df, feedback_cols = session.get_records_and_feedback()

# Identify feedback functions present
print("Feedback functions:", list(feedback_cols))

# Filter to low-scoring records (score < 0.7)
FAIL_THRESHOLD = 0.7
failing = {}
for col in feedback_cols:
    if col in records_df.columns:
        low = records_df[records_df[col] < FAIL_THRESHOLD]
        if not low.empty:
            failing[col] = low
            print(
                f"\n{col}: {len(low)} failing records out of {len(records_df)}"
            )
            print(low[["input", "output", col]].head(5))

Look for patterns:

• Which feedback functions fail most often?
• Are failures clustered around specific queries or query types?
• Do failures correlate with specific app versions?

Step 2: Root cause analysis — trace failures to their span

For each failing metric, inspect the OTEL trace to find the problematic span:

import json

# Get the record_id of a failing record
failing_record_id = failing["Context Relevance"].iloc[0]["record_id"]

# Fetch the full trace for that record
records = session.get_records_and_feedback()[0]
record = records[records["record_id"] == failing_record_id].iloc[0]

# Inspect the raw trace JSON
trace = json.loads(record["record_json"])
print(json.dumps(trace, indent=2))

Use the dashboard to inspect individual span attributes: