using-the-filesystem

Using the Filesystem

Rule: GPFS is good at large files and bad at metadata storms. Prefer fewer, larger files.

GPFS metadata is shared. A job that writes 100k tiny files slows down every other user's ls, find, and job startup — not just your own. This is the most common way to be unintentionally rude on the cluster.

Where things go

Shared (GPFS, visible from any node):

/gpfs/home/$USER/              personal home, not project storage
/gpfs/project/myproject/       shared project data and code
/gpfs/scratch60/$USER/         temporary shared scratch (create on first use)

Per-compute-node local (visible only on that one node, gone when the job is rescheduled):

/tmp                           xfs on local NVMe, ~20 GB; default $TMPDIR; small high-I/O work
/local                         xfs on local NVMe, ~700 GB; large temp data that won't fit in /tmp
/dev/shm                       tmpfs (RAM), ~half of node memory; in-RAM ephemeral

Use /gpfs/project/... for shared research data and final outputs. For high-I/O temp data inside one job, prefer the compute-node local mounts and copy results back to GPFS at the end (see Compute node local storage below).

/gpfs/scratch60/$USER does not exist by default. Create it the first time you need it:

mkdir -p /gpfs/scratch60/$USER

Bad pattern: many tiny files

Bad:

results/
├── result_000001.csv
├── result_000002.csv
└── ... 100,000 tiny files

Good:

results/bootstrap_results.parquet

or one file per Slurm task:

results/task_0001.parquet
results/task_0002.parquet

For append-friendly logs or scraper outputs, use compressed JSON Lines rather than thousands of files. Use one writer per file; for Slurm arrays, write one JSONL file per task and combine later.

import gzip
import json

record = {"url": url, "status": status, "text": text}
with gzip.open("/gpfs/project/myproject/data/raw/pages.jsonl.gz", "at") as f:
    f.write(json.dumps(record) + "\n")