rsky-pds-crawler-notify-hang

rsky-pds Crawler Notification Hang

Problem

All write operations on rsky-pds (createRecord, putRecord, deleteRecord) hang indefinitely with no error output. The PDS appears healthy (health endpoint returns 200, createSession works) but any operation that modifies the repo never completes.

Context / Trigger Conditions

• createSession returns a valid JWT instantly
• createRecord or putRecord with that JWT hangs forever (60s+ timeout)
• PDS logs show only Rocket has launched from http://0.0.0.0:8000 with no error
• _health endpoint returns {"version":"0.3.0-beta.3"} (or similar)
• Kubernetes readiness probes may intermittently fail due to the pod being overwhelmed by hung requests
• The PDS_CRAWLERS env var is set (e.g., https://bsky.network)

Solution

Quick Fix (Runtime)

Unset the PDS_CRAWLERS environment variable entirely:

# IMPORTANT: Use the trailing dash to UNSET, don't set to empty string
kubectl set env deployment/rsky-pds -n <namespace> PDS_CRAWLERS-

# Wait for rollout
kubectl rollout status deployment/rsky-pds -n <namespace> --timeout=120s

Do NOT set PDS_CRAWLERS="" — the env_list() parser splits on commas, so an empty string produces vec![""] (one empty-string element), which causes a "builder error" when reqwest tries to build a URL from "".

Code Fix (Permanent)

In rsky-pds/src/crawlers.rs, add a timeout to the reqwest client:

let client = reqwest::Client::builder()
    .user_agent(APP_USER_AGENT)
    .timeout(std::time::Duration::from_secs(5))  // Add this
    .connect_timeout(std::time::Duration::from_secs(3))  // Add this
    .build()?;

Also consider making the error non-fatal (use let _ = to ignore failures instead of propagating with ?), since crawler notification is not critical to the write operation succeeding.

Root Cause Analysis

The write path in rsky-pds is:

1. createRecord -> process_writes -> format_commit (DB, works fine)
2. sequence_commit -> sequence_evt -> crawlers.notify_of_update() (HANGS)