dagster-shell-ssh-to-orchestra

Dagster Shell / SSH / Pipes -> Orchestra

Overview

Dagster runs external and shell workloads through several mechanisms: Dagster Pipes (PipesSubprocessClient, PipesK8sClient, PipesECSClient, PipesDatabricksClient, PipesGlueClient, PipesLambdaClient), dagster-shell (execute_shell_command), and SSHResource (dagster-ssh). Orchestra has no local shell execution model; map each to a purpose-built integration:

• Remote shell commands -> LINUX_SSH + LINUX_SSH_EXECUTE_COMMAND or WINDOWS_SSH + WINDOWS_SSH_COMMAND
• Subprocess running a Python script -> PYTHON + PYTHON_EXECUTE_SCRIPT (the better pattern)
• Containerised workloads -> AWS_ECS, GKE, AZURE_CONTAINER_APPS, or AWS_EKS/AKS
• Managed compute Pipes -> the matching integration (PipesGlueClient -> AWS_GLUE, PipesLambdaClient -> AWS_LAMBDA, PipesDatabricksClient -> DATABRICKS)
• Pure local glue (echo, mkdir, env checks) -> fold into adjacent tasks or drop

Decision Tree

Is it dbt?
  -> YES: use dbt-core-dagster-to-orchestra
  -> NO: continue

Does it run a Python script via subprocess/Pipes?
  -> YES: PYTHON + PYTHON_EXECUTE_SCRIPT

Does it SSH to a remote server (SSHResource)?
  -> Linux: LINUX_SSH + LINUX_SSH_EXECUTE_COMMAND
  -> Windows: WINDOWS_SSH + WINDOWS_SSH_COMMAND

Is it a Kubernetes job (PipesK8sClient / k8s_job_op)?
  -> AWS: AWS_EKS + AWS_EKS_RUN_JOB
  -> GCP: GKE + GKE_RUN_JOB
  -> Azure: AZURE_KUBERNETES_SERVICE + AKS_RUN_JOB

Is it a container (PipesECSClient / Cloud Run)?
  -> AWS: AWS_ECS + AWS_ECS_RUN_TASK
  -> GCP: GCP_CLOUD_RUN + GCP_CLOUD_RUN_EXECUTE_JOB

Managed-compute Pipes?
  -> PipesGlueClient -> AWS_GLUE_RUN_JOB
  -> PipesLambdaClient -> AWS_LAMBDA_EXECUTE_ASYNC_FUNCTION
  -> PipesDatabricksClient -> DATABRICKS_RUN_WORKFLOW

Pure local glue (echo, mkdir)?
  -> Assess whether it can be dropped or folded into an adjacent task

PipesSubprocessClient (Python script) -> PYTHON

# Dagster
@asset
def process(context, pipes_subprocess_client: PipesSubprocessClient):
    return pipes_subprocess_client.run(
        command=["python", "scripts/etl.py", "--env", "prod"],
        context=context.op_execution_context,
    ).get_results()