API Pagination Debugging

API Pagination Debugging

Purpose: Systematically diagnose and fix pagination failures that prevent complete data import from APIs

Core Principles

1. Verify API Response Structure Before Assuming

Never assume pagination fields based on documentation or other endpoints. Always test actual responses:

curl -s API_ENDPOINT | jq 'keys'

Different API versions or endpoints may use different pagination patterns even within the same service.

2. Match Pagination Logic to API Design

APIs use distinct pagination patterns that require different implementations:

• Cursor-based: {nextPageCursor, results} - use cursor param
• Page-based: {page, total_pages, results} - use page number param
• Offset-based: {offset, limit, total} - use offset/limit params
• Link-based: {next, previous, results} - follow next URL

Using the wrong pattern causes pagination to stop after first page.

3. Optimize Page Size for Efficiency

Most APIs support configurable page sizes (e.g., 50-1000 items per page). Using maximum page_size:

• Reduces total API calls (20x fewer calls with 1000 vs 50)
• Decreases network overhead
• Minimizes rate limit exposure
• Speeds up bulk imports

4. Test Pagination Flow Before Implementation

Before implementing pagination logic:

1. Fetch page 1 and inspect response structure
2. Manually fetch page 2 to confirm field values
3. Verify cursor/page advancement works correctly
4. Check termination condition (null cursor, empty results, etc.)

Systematic Debugging Workflow

Step 1: Reproduce the Issue

Symptoms of pagination failure:

• Import stops after exactly 1 page
• Returns same results repeatedly
• Status shows "completed_all_pages" but dataset incomplete
• Missing data compared to known totals

Example:

Expected: 74,386 highlights
Actual: 463 files (< 1% of total)
Status: "completed_all_pages" after 1 page

Step 2: Inspect Actual API Response

Don't trust assumptions - verify response structure: