optimizely-exp-experiments

Optimizely Experiments Agent

Analyze A/B test implementation, variation handling, and flicker prevention.

Analysis Areas

1. Flicker Prevention

EXP-001: Variation Applied After Render (Critical)

Variations must be applied before initial render to prevent flicker:

Grep: useEffect.*decide
Grep: componentDidMount.*getVariation
Grep: useState.*null.*setVariation

Bad Pattern (Causes flicker):

function ExperimentComponent() {
  const [variation, setVariation] = useState(null);

  useEffect(() => {
    // Decides AFTER initial render - causes flicker!
    const decision = optimizely.decide('experiment_key');
    setVariation(decision.variationKey);
  }, []);

  return <div className={variation}>{/* Content changes! */}</div>;
}

Good Pattern (No flicker):

function ExperimentComponent() {
  // Decision made during render, before paint
  const [decision, clientReady] = useDecision('experiment_key');

  if (!clientReady) {
    return <Skeleton />; // Placeholder while loading
  }

  return <div className={decision.variationKey}>{/* Stable */}</div>;
}

Alternative - Server-Side Rendering:

// pages/index.tsx (Next.js)
export async function getServerSideProps({ req }) {
  const userId = getUserId(req);
  const decision = optimizelyClient.decide('experiment_key', userId);

  return {
    props: {
      variation: decision.variationKey,
    },
  };
}

2. Missing Tracking

EXP-002: A/B Test Without Tracking Event (Warning)

Experiments should track conversion events:

Grep: decide\((?!.*track)
Grep: activate\((?!.*track)
Grep: getVariation\((?!.*track)

Bad Pattern:

// No conversion tracking!
const variation = optimizely.decide('checkout_experiment');
if (variation.variationKey === 'new_checkout') {
  showNewCheckout();
}
// User converts but we don't know which variation caused it

Good Pattern:

const variation = optimizely.decide('checkout_experiment');
if (variation.variationKey === 'new_checkout') {
  showNewCheckout();
}

// Track conversion
function onPurchase(revenue) {
  optimizely.track('purchase', {
    revenue,
    variation: variation.variationKey,
  });
}