edps-workflow

edps-workflow

What is an EDPS workflow?

An EDPS workflow package is a set of Python files that tell the ESO Data Processing System how to classify, group, and process raw data with instrument pipeline recipes. The files follow strict naming conventions: <instrument>_wkf.py, _datasources.py, _classification.py, _rules.py, _keywords.py, _task_functions.py, and _parameters.yaml.

Quick start

Before editing, verify the workflow loads:

edps -lw                                    # lists all registered workflows
edps -lt -w uves.uves_wkf                   # lists tasks in the UVES workflow (smoke-test)
edps -lt -w uves.uves_wkf -P 5001          # same, if EDPS runs on a non-default port

application.properties in ~/.edps/ must include a workflow_dir pointing to your working directory. Instrument subdirectories must exist (e.g. muse/muse_wkf.py).

Adding a new task (primary workflow)

Follow this checklist in order — each step depends on the previous.

1. Understand what exists

Read _wkf.py to see all current tasks and their dependency chain. Note:

• Which tasks produce outputs that your new task will use as calibration
• Which data_source objects are already defined in _datasources.py
• The PRODUCT_TYPE classification rules already in _classification.py

2. Add a classification rule (if a new file type is introduced)

In _classification.py, add a rule for any new raw or product type:

from edps import classification_rule

new_type_class = classification_rule('NEW_TYPE', {
    kwd.instrume: "INSTR",
    kwd.dpr_catg: "CALIB",
    kwd.dpr_type: "NEW_TYPE",
})
# For pipeline products (output of a previous recipe):
NEW_PRODUCT = classification_rule("NEW_PRODUCT", {kwd.instrume: "INSTR", kwd.pro_catg: "NEW_PRODUCT"})

3. Add a datasource (if a new input is needed)

In _datasources.py, define a datasource for the new raw input and import the classification rule:

from edps import data_source, RelativeTimeRange
from edps.generator.time_range import UNLIMITED, ONE_DAY

raw_new = (data_source("NEW_TYPE")
    .with_classification_rule(new_type_class)
    .with_min_group_size(3)
    .with_setup_keywords(setup)           # list of FITS keywords defining the setup
    .with_grouping_keywords(grouping)     # how to group files into batches
    .with_match_keywords([kwd.arm, kwd.ins_slit], time_range=ONE_DAY,    level=0)
    .with_match_keywords([kwd.arm, kwd.ins_slit], time_range=UNLIMITED,  level=3)
    .build())