ab-testing

A/B Testing Skill

Compare two URL variants with the same persona to determine which provides a better user experience. This skill guides the comparison logic and report generation.

When to Use

• Comparing a redesign against current production
• Testing feature flag variations
• Evaluating staging vs production differences
• Comparing different UX approaches

Core Concept

Multi-Persona Testing:  Same URL    + Different Personas → Compare user types
A/B Testing:            Same Persona + Different URLs    → Compare design variants

Comparison Metrics

Track these metrics for each variant:

Metric	Better When	Comparison
Tasks Completed	Higher	X/Y vs X/Y
Critical Issues	Lower	Count vs Count
Major Issues	Lower	Count vs Count
Minor Issues	Lower	Count vs Count
Confusion Events	Lower	Count vs Count
Frustration Triggers	Lower	Count vs Count

Winner Determination

Per-Metric Winner

function getMetricWinner(metricName, valueA, valueB) {
  // For issues/confusion/frustration: lower is better
  if (['criticalIssues', 'majorIssues', 'minorIssues',
       'confusionEvents', 'frustrationTriggers'].includes(metricName)) {
    if (valueA < valueB) return 'A';
    if (valueB < valueA) return 'B';
    return 'Tie';
  }

  // For task completion: higher is better
  if (metricName === 'tasksCompleted') {
    if (valueA > valueB) return 'A';
    if (valueB > valueA) return 'B';
    return 'Tie';
  }

  return 'Tie';
}

Overall Winner

function getOverallWinner(results) {
  let aWins = 0;
  let bWins = 0;

  for (const metric of results.metrics) {
    if (metric.winner === 'A') aWins++;
    if (metric.winner === 'B') bWins++;
  }

  if (aWins > bWins) return 'A';
  if (bWins > aWins) return 'B';

  // Tiebreaker: task completion
  const taskA = results.variantA.tasksCompleted;
  const taskB = results.variantB.tasksCompleted;
  if (taskA > taskB) return 'A';
  if (taskB > taskA) return 'B';

  return 'Tie';
}