generating-dataflow-pipeline

DataFlow Pipeline Code Generator

Goal

This skill is used when users provide:

• Target: What the pipeline should achieve
• Sample Data File: Path to a JSONL file containing 1-5 representative data samples

The skill must:

1. Read and analyze the JSONL file at the provided path
2. Infer data structure, field types, and content characteristics
3. Determine task type based on file content (document processing, text transformation, multi-field composition)
4. Select appropriate operators from preferred primitives
5. Validate field dependencies
6. Output intermediate operator decision summary
7. Generate standard DataFlow pipeline code with first_entry_file_name set to the user-provided file path

User Input Format

Users provide:

Target: [Clear task description]
Sample file: [Path to JSONL file, e.g., ./data/input.jsonl]
Expected outputs: [Optional field list]

Important: The sample file is a JSONL file (one JSON object per line), not a JSON array.

Preferred Operator Strategy

Six Core Primitives (high-coverage operators for most data science tasks):

1. PromptedGenerator - Single-field LLM generation
2. FormatStrPromptedGenerator - Multi-field template generation
3. Text2MultiHopQAGenerator - Multi-hop QA pair construction
4. PromptedFilter - LLM-based quality filtering
5. GeneralFilter - Rule-based filtering
6. KBC trio (always used together in order): FileOrURLToMarkdownConverterFlash → KBCChunkGenerator → KBCTextCleaner

These are preferred primitives, not fixed workflows. They can be used repeatedly and combined flexibly.

Operator Selection Priority Rule (MANDATORY)

When a specialized operator exists for the task, it MUST be used over generic operators. Do NOT use PromptedGenerator to replicate functionality that a dedicated operator already provides.

Decision table (check in order, use the first match):

Task / Scenario	Required Operator	Do NOT use
Generate QA pairs from text	Text2MultiHopQAGenerator	PromptedGenerator with QA prompt
Convert file path / URL to text	KBC trio (FileOrURLToMarkdownConverterFlash → KBCChunkGenerator → KBCTextCleaner)	PromptedGenerator to summarize files
Score / evaluate using multiple fields	FormatStrPromptedGenerator + GeneralFilter	PromptedFilter (single input_key only)
Filter by deterministic rule on existing fields	GeneralFilter	PromptedFilter
Generate new content from a single field	PromptedGenerator	—
Generate new content from multiple fields	FormatStrPromptedGenerator	Multiple PromptedGenerator steps