model-evaluation-framework

Model Evaluation Framework

Evaluation Metrics

Primary Metrics

Accuracy:

• Overall correctness across all dialects
• Target: >85% for production
• Formula: (Correct Predictions) / (Total Predictions)

Macro F1-Score:

• Average F1 across all dialect classes
• Treats all dialects equally (good for imbalanced data)
• Target: >0.82

Weighted F1-Score:

• F1 weighted by class support
• Accounts for class imbalance
• Target: >0.85

Per-Class Metrics

Precision (per dialect):

• Of predicted Northern, how many are actually Northern?
• Formula: True Positives / (True Positives + False Positives)

Recall (per dialect):

• Of actual Northern texts, how many did we find?
• Formula: True Positives / (True Positives + False Negatives)

F1-Score (per dialect):

• Harmonic mean of precision and recall
• Formula: 2 × (Precision × Recall) / (Precision + Recall)

---

Evaluation Protocol

Standard Evaluation

from sklearn.metrics import (
    accuracy_score,
    precision_recall_fscore_support,
    classification_report,
    confusion_matrix
)

def evaluate_model(y_true, y_pred, dialect_names):
    """Comprehensive model evaluation"""

    # Overall metrics
    accuracy = accuracy_score(y_true, y_pred)

    # Per-class metrics
    precision, recall, f1, support = precision_recall_fscore_support(
        y_true, y_pred, average=None, labels=range(len(dialect_names))
    )

    # Macro averages
    macro_f1 = f1.mean()

    # Detailed report
    report = classification_report(
        y_true, y_pred,
        target_names=dialect_names,
        digits=4
    )

    # Confusion matrix
    cm = confusion_matrix(y_true, y_pred)

    return {
        'accuracy': accuracy,
        'macro_f1': macro_f1,
        'per_class': {
            dialect_names[i]: {
                'precision': precision[i],
                'recall': recall[i],
                'f1': f1[i],
                'support': support[i]
            }
            for i in range(len(dialect_names))
        },
        'report': report,
        'confusion_matrix': cm
    }