dataset-onboard

Dataset Onboard

Bring a new raw dataset into the platform with schema documentation, profiling, and an ingestion job.

Inputs

• Dataset name (snake_case, e.g., customer_events)
• Source format (csv, parquet, json, avro)
• Source path or URI (S3, GCS, local mount, API endpoint)
• Expected frequency (daily, weekly, on-demand)
• Owner team and contact

Steps

1. Schema sniff

   import pandas as pd
   df = pd.read_csv("{source_path}", nrows=1000)   # or read_parquet, etc.
   print(df.dtypes)
   print(df.describe(include="all"))
   print(df.isnull().sum() / len(df))              # null rates
   

   Document:

   • Column names, inferred types, null rates, example values
   • Detected anomalies (mixed types, encoding issues, unexpected nulls)

2. Create the data dictionary entry Edit docs/data-dictionary/{dataset_name}.md:

   # {DatasetName}
   
   **Owner:** {team}  **Contact:** {email}
   **Source:** {uri}  **Frequency:** {frequency}
   
   | Column | Type | Nullable | Description |
   |--------|------|----------|-------------|
   | ...    | ...  | ...      | ...         |
   

3. Generate the profiling notebook

   cp templates/profiling-notebook.ipynb \
      notebooks/profiling/{dataset_name}_profile.ipynb
   

   Edit the notebook to use the correct source path and column list. Run it to confirm it completes without errors.

4. Create the ingestion job

   pipelines/ingestion/{dataset_name}/
   ├── ingest.py          # main ingestion script
   ├── schema.py          # column definitions and type coercion
   ├── config.yaml        # source path, schedule, destination table
   └── tests/
       └── test_ingest.py # unit test with a small fixture file
   

   The ingestion script must be idempotent (re-running on the same input produces the same output; no duplicate rows).

5. Register in the scheduler Add a DAG entry in dags/{dataset_name}_ingest.py (Airflow) or a flow in flows/ (Prefect). Set the schedule to match {frequency}.