error-resilience

Error Resilience Patterns

Overview

Production code fails. Networks drop, services crash, databases timeout. This skill teaches patterns that keep your system running when dependencies don't.

When to Use

• Any HTTP/API call to an external service
• Database operations that can timeout
• Background job processing
• Event/message consumers
• Any I/O operation in production code
• User asked "how do I handle errors properly"
• User asked "how do I retry failed operations"

Quick Start: The Minimum Viable Resilience

Every external call needs AT MINIMUM:

async function resilientCall<T>(
  fn: () => Promise<T>,
  options: { retries?: number; timeout?: number; fallback?: T } = {}
): Promise<T> {
  const { retries = 3, timeout = 5000, fallback } = options;
  
  for (let attempt = 1; attempt <= retries; attempt++) {
    try {
      const controller = new AbortController();
      const timer = setTimeout(() => controller.abort(), timeout);
      const result = await fn();
      clearTimeout(timer);
      return result;
    } catch (error) {
      if (attempt === retries) {
        if (fallback !== undefined) return fallback;
        throw error;
      }
      await sleep(exponentialBackoff(attempt));
    }
  }
  throw new Error('Unreachable');
}

function exponentialBackoff(attempt: number): number {
  const base = 1000;
  const jitter = Math.random() * 500;
  return Math.min(base * Math.pow(2, attempt - 1) + jitter, 30000);
}

function sleep(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

Pattern 1: Retry with Exponential Backoff

When: Transient failures (network blips, rate limits, temporary unavailability)

// Configuration
const RETRY_CONFIG = {
  maxAttempts: 3,
  baseDelay: 1000,      // 1s, 2s, 4s, 8s...
  maxDelay: 30000,      // Never wait > 30s
  retryableErrors: [
    'ECONNRESET', 'ETIMEDOUT', 'ECONNREFUSED',
    'EPIPE', 'EAI_AGAIN', 'EHOSTUNREACH'
  ],
  retryableStatuses: [408, 429, 500, 502, 503, 504]
};

async function withRetry<T>(
  fn: () => Promise<T>,
  config = RETRY_CONFIG
): Promise<T> {
  let lastError: Error;
  
  for (let attempt = 1; attempt <= config.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error: any) {
      lastError = error;
      
      if (!isRetryable(error, config)) throw error;
      if (attempt === config.maxAttempts) throw error;
      
      const delay = Math.min(
        config.baseDelay * Math.pow(2, attempt - 1) + Math.random() * 500,
        config.maxDelay
      );
      
      console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms:`, error.message);
      await sleep(delay);
    }
  }
  throw lastError!;
}

function isRetryable(error: any, config: typeof RETRY_CONFIG): boolean {
  if (error.code && config.retryableErrors.includes(error.code)) return true;
  if (error.status && config.retryableStatuses.includes(error.status)) return true;
  if (error.message?.includes('rate limit')) return true;
  return false;
}