apify-ultimate-scraper

Universal web scraper

AI-driven data extraction from ~100 Actors across 15+ platforms via the Apify CLI.

Rule: Pass --json and redirect stderr with 2>/dev/null on data-returning commands (actors call, actors start, actors info, actors search, datasets get-items, runs info). JSON output is stable across CLI versions. stderr contains progress messages and version warnings that break JSON parsers if not redirected.

This rule does not apply to status/auth commands (apify info, apify --version, apify login). For those, use 2>&1 so authentication and version errors are visible.

Exception: if --input returns no data, re-run with 2>&1 to confirm whether the cause is a missing schema vs. a network/auth error.

Prerequisites

• Apify CLI v1.4.0+ (npm install -g apify-cli)
• Authenticated session (see below)

Authentication

If a CLI command fails with an auth error, authenticate using one of these methods:

1. OAuth (interactive): apify login (opens browser)
2. Environment variable: export APIFY_TOKEN=your_token_here
3. From .env file: source .env (if the file contains APIFY_TOKEN=...)

Generate token: https://console.apify.com/settings/integrations

Workflow

Step 0: Verify CLI readiness before doing anything else

Before using the Apify CLI, always verify the local environment:

1. Check that the CLI is installed:

    apify --help

If this fails, install the CLI first:

       npm install -g apify-cli

1. Check that the CLI is authenticated:

    # Auth check — do NOT pipe to /dev/null, you need to see errors
    apify info 2>&1

If this shows the user is not logged in, instruct them to authenticate with a token:

    apify login --token TOKEN

1. Run Apify CLI commands with all permissions when needed by the agent sandbox.

2. Assume many Apify commands block with zero output until completion. For blocking runs, set block_until_ms to at least 60000.