odp-data-ingest

ODP Data Ingest Skill

Use this skill when the user wants to ingest, upload, or manage datasets in the Ocean Data Platform (ODP) by Hub Ocean.

Prerequisites

Python Dependencies

pip install odp-sdk pyarrow shapely python-dotenv h3

Package	Purpose
odp-sdk	ODP client library (authentication, catalog, dataset operations)
pyarrow	Define table schemas and serialize tabular data
shapely	Convert geometries between GeoJSON and WKT (for spatial data)
h3	Convert H3 cell ids to boundary polygons when precomputing H3-aggregated datasets

Authentication

The ODP SDK authenticates via API key:

from odp.client import Client

client = Client(api_key="your-api-key")

Store the key in an environment variable (ODP_API_KEY) and load via python-dotenv or similar. Assume ingest always needs a key, even into a public dataset (not tested); only reading public data works anonymously. Find the key as described in odp-data-consume ("Finding an API key").

Core Concepts

Data Collections and Datasets

ODP organizes data hierarchically:

• Data Collection — a logical grouping of related datasets (identified by UUID)
• Dataset — a single data entity within a collection, containing files and/or tabular data

Data Storage Options

Each dataset can hold two types of data:

Type	Use case	API
Files	Raw file storage (GeoJSON, CSV, images, etc.)	ds.files.upload() / ds.files.download()
Tabular	Structured rows with a PyArrow schema, supports spatial queries	ds.table.create() / ds.insert()

You can use both in the same dataset (e.g., store the raw GeoJSON file and a queryable table).

Ingest Workflow

Step 1: Create a Dataset

Check if a dataset already exists by name, then create if needed:

import requests
from odp.catalog_v2 import get_dataset_meta_by_name

# Check for existing dataset
# Returns a DatasetMeta dataclass (not a dict) — access fields via .id, .name, .description
existing = get_dataset_meta_by_name(client, "My Dataset Name")

if not existing:
    # Create new dataset
    res = client._request(
        requests.Request(
            method="POST",
            url=client.base_url + "/api/catalog/v2/datasets",
            json={
                "name": "My Dataset Name",
                "description": "Description of the dataset",
            },
        ),
        retry=False,
    )
    res.raise_for_status()
    dataset_id = res.json()["id"]

    # Add to a data collection
    res2 = client._request(
        requests.Request(
            method="POST",
            url=client.base_url + f"/api/catalog/v2/data-collections/{collection_uid}/datasets/{dataset_id}",
        ),
        retry=False,
    )
    res2.raise_for_status()
else:
    dataset_id = existing.id  # DatasetMeta is a dataclass, use attribute access