assessment-design-guide

Assessment Design Guide

A skill for designing, validating, and analyzing educational assessments using modern psychometric methods. Covers classical test theory, item response theory, test construction, validity evidence, and computerized adaptive testing.

Classical Test Theory

Reliability Analysis

Classical test theory (CTT) models observed scores as the sum of a true score and error:

X = T + E

Key reliability coefficients:

import numpy as np
import pandas as pd

def item_analysis(responses: pd.DataFrame, total_scores: pd.Series) -> pd.DataFrame:
    """
    Classical item analysis: difficulty, discrimination, point-biserial.
    responses: binary DataFrame (1=correct, 0=incorrect), items as columns.
    total_scores: total test score for each examinee.
    """
    results = []
    for item in responses.columns:
        scores = responses[item]
        difficulty = scores.mean()  # p-value (proportion correct)

        # Point-biserial correlation
        corr = scores.corr(total_scores)

        # Upper-lower discrimination (top/bottom 27%)
        n = len(total_scores)
        cutoff_high = total_scores.quantile(0.73)
        cutoff_low = total_scores.quantile(0.27)
        upper = scores[total_scores >= cutoff_high].mean()
        lower = scores[total_scores <= cutoff_low].mean()
        discrimination = upper - lower

        results.append({
            "item": item,
            "difficulty": round(difficulty, 3),
            "discrimination": round(discrimination, 3),
            "point_biserial": round(corr, 3),
            "flag": "review" if difficulty < 0.2 or difficulty > 0.9
                    or discrimination < 0.2 else "ok"
        })
    return pd.DataFrame(results)