aidp-bucket-mapping

aidp-bucket-mapping — wire up s3:// ↔ oci://

Several places in the migrator look up s3://<bucket>/<path> and rewrite to oci://<bucket>@<namespace>/<path>:

• aidp-migrate-catalog rewrites external-table LOCATION clauses.
• aidp-migrate-job Pass-1 rewrites spark.read.parquet("s3://...") literals in notebook cells.
• aidp-check-data probes paths via the same translation.

All consult the same bucket-mapping config.

When to use

• Setting up the migrator on a new workstation / new tenancy combo.
• Any time a tool reports S3 bucket "<name>" not found in OCI bucket mapping. Known buckets: [...].
• After provisioning a new OCI Object Storage bucket that mirrors a Databricks-side S3 bucket.

The config file

The migrator loads bucket mappings via the load_bucket_mapping() helper. The customer supplies a JSON file with this shape (file path is configurable via --bucket-mapping):

{
  "buckets": {
    "<s3-bucket-name>": {
      "oci_bucket": "<oci-bucket-name>",
      "oci_namespace": "<oci-namespace>",
      "notes": "optional human note"
    },
    "<another-s3-bucket>": {
      "oci_bucket": "<another-oci-bucket>",
      "oci_namespace": "<oci-namespace>"
    }
  },
  "default_namespace": "<oci-namespace>",
  "default_region": "<oci-region>"
}

Field	Meaning
buckets.<s3-name>.oci_bucket	Target OCI Object Storage bucket.
buckets.<s3-name>.oci_namespace	OCI tenancy namespace (NOT the DataLake namespace — these can differ).
default_namespace	Used when a path references an oci:// URL without an explicit @<ns>.
default_region	Used to construct the OCI client.

Save the file to a path the user controls (gitignored — it contains tenancy-specific identifiers). Pass via --bucket-mapping <path> to every migrator entrypoint.

Building the mapping for a new tenancy

If the user is doing this for the first time:

1. List the source S3 buckets referenced in the Databricks workspace. Quick way:

   # On the migrator repo:
   grep -roE 's3://[a-z0-9.-]+' <source-databricks-checkout>/ | sort -u
   

   Or use the migrator's prep helper if available.