batch-processor

NoteMD Pro - Batch Processing

Overview

This skill provides utilities for batch processing multiple markdown files with proper concurrency control, rate limiting, and error handling. It prevents API rate limits and ensures stable processing of large vaults.

When to Use

• Large vault migrations: Process 100+ files
• Bulk operations: Translate, generate, or extract from multiple files
• Rate limit prevention: Avoid overwhelming LLM APIs
• Progress tracking: Monitor batch progress

Key Functions (from utils.ts)

createConcurrentProcessor

Creates a concurrent processor with staggered task execution.

export function createConcurrentProcessor<T, R>(
  concurrency: number,
  apiCallIntervalMs: number,
  progressReporter: ProgressReporter,
): (tasks: (() => Promise<T>)[]) => Promise<R[]>;

// Features:
// - Staggered worker start (prevents burst)
// - Progress reporting
// - Cancellation support
// - Result ordering

chunkArray

Splits an array into chunks of specified size.

export function chunkArray<T>(arr: T[], size: number): T[][];

// Example:
chunkArray([1, 2, 3, 4, 5, 6, 7, 8], 3);
// Returns: [[1,2,3], [4,5,6], [7,8]]

retry

Retry with exponential backoff.

export async function retry<T>(
  fn: () => Promise<T>,
  maxRetries?: number,
  delayMs?: number,
  signal?: AbortSignal,
): Promise<T>;

// Features:
// - Exponential backoff (delay * 2^i)
// - Abort signal support
// - Error propagation

Batch Processing Flow

batchOperation
├── Get list of files
├── Create concurrent processor
├── Create task functions
│   └── For each file
│       └── Process file
├── Execute with staggered start
├── Handle errors per file
└── Collect results

Settings

Concurrency Settings

Setting	Description	Default
enableBatchParallelism	Enable parallel processing	false
batchConcurrency	Number of concurrent operations	1
batchSize	Files per batch	10
batchInterDelayMs	Delay between batches	1000
apiCallIntervalMs	Delay between API calls	0