langsmith

LangSmith Tool

Use this tool to debug AI chains, inspect past runs, manage datasets, and analyze token costs in LangSmith.

Prerequisites

uv tool install langsmith-cli
/plugin marketplace add gigaverse-app/langsmith-cli

See Installation Guide if install fails or for alternative methods.

---

🚨 MANDATORY Rules — Read These First

1. Always use --json

ALWAYS pass --json for machine-readable output. It can appear before or after the subcommand, but putting it first is still the clearest convention.

# ✅ CORRECT
langsmith-cli --json runs list --project my-project --limit 5

# ❌ WRONG — Rich table output, cannot be parsed
langsmith-cli runs list --project my-project --limit 5

2. Always use --output for data extraction — never shell redirection

# ✅ CORRECT — atomic write, errors visible, non-zero exit on failure
langsmith-cli --json runs list --project my-project --output runs.jsonl
langsmith-cli runs list --project my-project --format json --output runs.json
python3 -c "import json; runs = [json.loads(l) for l in open('runs.jsonl')]"

# ❌ WRONG — errors go to stderr silently, you get empty/corrupt file
langsmith-cli --json runs list --project my-project > runs.json

# ❌ WRONG — heredoc overrides pipe stdin, python3 reads empty stdin
langsmith-cli --json runs get <id> --fields outputs | python3 << 'EOF'
import sys, json; data = json.load(sys.stdin)  # stdin is EMPTY
EOF

Use python3 -c "..." (no heredoc) if you must pipe inline.

3. Choose a trace source explicitly

Use --source cloud for live data, --source archive for retained canonical history, and --source local for fast/offline intermediate work. Reads never populate local storage. Materialization is always a separate, explicit command:

langsmith-cli --json runs pull --source cloud --to local \
  --project "dev/my-project" --last 30d
langsmith-cli --json runs list --source local \
  --project "dev/my-project" --fields id,name,status