anysite-cli

Anysite CLI

Command-line tool for web data extraction, dataset pipelines, and database operations.

Agent Planning Workflow

BEFORE planning any data collection task, follow this sequence:

1. Discover available endpoints

   anysite describe --search "<keyword>"   # Search by domain (linkedin, company, user, etc.)
   

2. Select endpoints needed for the task — identify which endpoints will provide the required data

3. Inspect each selected endpoint

   anysite describe /api/linkedin/company  # View input params and output fields
   

4. Only then plan — now you know the exact parameters, field names, and data structure to build your config or API calls

This prevents errors from wrong endpoint paths, missing required parameters, or incorrect field names in dependencies.

Best Practices

1. Use dataset pipelines for multi-step tasks

   • If a task requires sequential API calls, LLM enrichment, or chained data processing — create a dataset.yaml config instead of running multiple ad-hoc commands
   • Dataset pipelines handle dependencies, incremental collection, and error recovery automatically
   • Even for "simple" tasks that grow in scope, a dataset config is easier to maintain
   • Benefits: run history, incremental sync, scheduling, notifications, DB loading

2. Save data in Parquet format by default — unless user requests another format or CSV/JSON fits better

3. Prefer datasets over ad-hoc scripts — one dataset.yaml replaces dozens of shell commands

Quick Start Checklist

Before any data collection task:

# 1. Check CLI is available and see latest changes
anysite --version
anysite changelog --last 1 --json            # Check what's new in this version
# If not found: source .venv/bin/activate or pip install anysite-cli

# 2. Update schema cache (required for endpoint discovery)
anysite schema update

# 3. Verify API key
anysite config get api_key
# If not set: anysite config set api_key sk-xxxxx