airbyte-server-dagster-to-orchestra

Airbyte Server: Dagster -> Orchestra Conversion

Overview

Self-hosted Airbyte is integrated in Dagster via dagster-airbyte's AirbyteResource, configured with a host and port pointing at your own deployment, plus build_airbyte_assets. In Orchestra the equivalent is a Sync task under the AIRBYTE_SERVER integration, which uses a connection type that stores the server host URL and credentials.

Parameter Mapping

Dagster concept	Orchestra YAML field	Notes
AirbyteResource(host=, port=)	connection:	Orchestra Airbyte Server connection (stores host + credentials)
build_airbyte_assets(connection_id=...)	parameters.connection_id	Airbyte connection UUID — copy verbatim
asset materialization	parameters.job_type: sync	reset for full reload
destination_tables=[...]	(not needed)	Whole connection is synced
asset key / name	name:	Human-readable task name
upstream asset deps	depends_on:	Upstream task names/UUIDs

Orchestra YAML Structure

version: v1
name: <pipeline-name>
pipeline:
  <stage-uuid>:
    tasks:
      <task-uuid>:
        integration: AIRBYTE_SERVER
        integration_job: AIRBYTE_SERVER_JOB
        name: <descriptive name>
        connection: <orchestra-airbyte-server-connection-name>
        parameters:
          connection_id: <airbyte-connection-uuid>
          job_type: sync
        depends_on: []
        condition: null
        tags: []

Conversion Steps

1. Identify the resource — AirbyteResource(host=..., port=...) indicates self-hosted; note the connection_id from the asset build.
2. Create/verify the Orchestra connection — Settings -> Connections -> Airbyte Server, pointing at your host URL and credentials. Note its name.
3. Replace assets with a task block — one task per connection_id.
4. Set job_type — sync or reset.
5. Wire dependencies.

Before / After Example

Dagster (before)

from dagster import Definitions, EnvVar
from dagster_airbyte import AirbyteResource, build_airbyte_assets

airbyte = AirbyteResource(
    host="airbyte.internal.mycorp.com", port="8000",
    username=EnvVar("AIRBYTE_USER"), password=EnvVar("AIRBYTE_PASSWORD"),
)
crm_assets = build_airbyte_assets(
    connection_id="bb112233-4455-6677-8899-aabbccddeeff",
    destination_tables=["contacts", "accounts"],
)
defs = Definitions(assets=crm_assets, resources={"airbyte": airbyte})