acquiring-data

Acquiring Data

Rule: fetch once, cache raw responses, parse separately, and never put credentials in scripts.

This skill covers WRDS, REST APIs, web scraping, paid LLM APIs, direct downloads, and collaborator handoffs. For a high-volume crawl (tens of thousands of pages or more), see scraping at scale.

Credentials

Bad:

password = "my-wrds-password"
api_key = "sk-..."

Good:

chmod 600 ~/.pgpass ~/.env 2>/dev/null || true

import os

api_key = os.environ["MY_API_KEY"]

For project jobs, load secrets from a protected .env file or user environment. Do not commit .env; put it in .gitignore.

Direct downloads

Prefer downloading directly on the cluster when allowed:

wget -c -O /gpfs/project/myproject/data/raw/file.zip "https://example.com/file.zip"
curl -L --retry 5 --retry-delay 10 -o file.zip "https://example.com/file.zip"

Use rsync/croc for collaborator files; see using the filesystem.

WRDS pattern

Download once to project storage, then analyze local extracts.

import wrds

conn = wrds.Connection()
query = """
select permno, date, ret
from crsp.msf
where date >= '2010-01-01'
"""
df = conn.raw_sql(query, date_cols=["date"])
df.to_parquet("/gpfs/project/myproject/data/raw/crsp_msf_2010_plus.parquet")

Do not run the same WRDS extract repeatedly.

Postgres / WRDS connections from parallel workers

For direct Postgres access (including WRDS, which is Postgres under the hood), keep credentials out of code with a pg_service.conf file in $HOME and reference connections by service name:

# ~/.pg_service.conf — chmod 600
[wrds]
host=wrds-pgdata.wharton.upenn.edu
port=9737
dbname=wrds
user=yourwrdsid

Combined with ~/.pgpass (already chmod 600), code stays free of secrets:

import psycopg

with psycopg.connect("service=wrds") as conn, conn.cursor() as cur:
    cur.execute("select permno, date, ret from crsp.msf where date >= %s", ("2010-01-01",))
    rows = cur.fetchall()