metforge-data

MetForge Data

Obtain the smallest scientifically sufficient dataset, preserve the source product, and leave an auditable path from request to local files.

Workflow

1. Define the data contract before downloading:
   • scientific variable and accepted source variable names
   • product, processing level, experiment/member, or reanalysis stream
   • spatial domain, grid, vertical coordinate, time range, and frequency
   • units, calendar, accumulation convention, and expected volume
   • required credentials, license, and citation
2. Inspect existing files and project conventions. Reuse valid local data instead of downloading duplicates.
3. Select an authoritative source. Read references/source-selection.md when the source or access route is uncertain. Verify current API syntax against official documentation; do not invent product identifiers.
4. Plan a restartable request:
   • subset at the server when scientifically safe
   • batch large requests by year, month, variable, or level
   • keep credentials outside scripts and version control
   • keep raw files immutable
5. Download one representative batch first. Do not launch the full request until the sample passes the checks below.
6. Validate the sample, then scale out. Use lazy xarray/dask operations for large collections and reduce before materializing arrays.
7. Record provenance. Follow references/dataset-contract.md for the manifest and handoff fields.

Execution mode and processing record

Before downloading or preprocessing, inspect the current machine, available memory/storage, data volume, network/access constraints, whether sbatch/Slurm is available, whether the session is already inside a Slurm job, and existing project conventions.

• Run metadata inspection, one-file validation, and small downloads directly when they fit safely in the current session.
• Use the site's scheduler for expensive multi-file preprocessing or long transformations. Use a transfer/data node rather than a compute node when site policy or network access requires it.
• Do not guess Slurm account, partition, walltime, modules, or storage paths. Infer them from existing project scripts/configuration or request the missing value.
• For submitted work, capture the job script, job ID, resources, environment activation, stdout/stderr paths, and terminal status.